Method and apparatus for processing video signal on basis of secondary transform

ABSTRACT

Disclosed is a method for decoding a video signal, including: generating a dequantized transform block by performing dequantization for a current block; acquiring an intra prediction mode of the current block; determining a secondary transform set applied to the current block among a plurality of secondary transform sets based on the intra prediction mode; deriving a transform kernel applied to the current block in the determined secondary transform set; and performing secondary transform for a top left specific area of the dequantized transform block by using the derived transform kernel, in which the plurality of secondary transform sets may include at least one mixed secondary transform set.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the Bypass Continuation of International ApplicationNo. PCT/KR2019/008060, filed on Jul. 2, 2019, which claims the benefitof U.S. Provisional Applications No. 62/693,394, filed on Jul. 2, 2018,and No. 62/693,397 filed on Jul. 2, 2018, the contents of which are allhereby incorporated by reference herein in their entirety.

TECHNICAL FIELD

The present disclosure relates to a method and an apparatus forprocessing a video signal, and more particularly, to a method fordesigning and applying secondary transform.

BACKGROUND ART

Next-generation video contents will have features such as high spatialresolution, high frame rate, and high dimensionality of scenerepresentation. In order to process the contents, a tremendous increasewill be caused in terms of memory storage, memory access rate, andprocessing power.

Therefore, there is a need to design a new coding tool for processingthe next-generation video contents more efficiently. In particular, whentransform is applied, there is a need to design more efficient transformin terms of coding efficiency and complexity.

DISCLOSURE Technical Problem

An embodiment of the present disclosure provides a method forconfiguring a secondary transform set by considering various conditionsin order to apply secondary transform.

Furthermore, an embodiment of the present disclosure also provides amethod for efficiently encoding/decoding a secondary transform indexsignaled from an encoder in configuring a secondary transform set basedon an intra prediction mode and/or a size of a coding block (ortransform block).

The technical objects of the present disclosure are not limited to theaforementioned technical objects, and other technical objects, which arenot mentioned above, will be apparently appreciated by a person havingordinary skill in the art from the following description.

Technical Solution

In an aspect of the present disclosure, provided is a method fordecoding a video signal, including: generating a dequantized transformblock by performing dequantization for a current block; acquiring anintra prediction mode of the current block; determining a secondarytransform set applied to the current block among a plurality ofsecondary transform sets based on the intra prediction mode; deriving atransform kernel applied to the current block in the determinedsecondary transform set; and performing secondary transform for a topleft specific area of the dequantized transform block by using thederived transform kernel, in which the plurality of secondary transformsets may include at least one mixed secondary transform set.

Preferably, the mixed secondary transform set may include at least one8×8 transform kernel applied to an 8×8 size area and at least one 4×4transform kernel applied to a 4×4 size area.

Preferably, when the plurality of secondary transform sets include aplurality of mixed secondary transform sets, the plurality of mixedsecondary transform sets may include different numbers of transformkernels, respectively.

Preferably, the deriving of the transform kernel may further includeacquiring a secondary transform index indicating a transform kernelapplied to the current block in the determined secondary transform set,and the secondary transform index may be binarized by a truncated unaryscheme based on a maximum number of available transform kernels in thedetermined secondary transform set.

Preferably, the determining of the secondary transform set may include:determining whether to use the mixed secondary transform set based onthe intra prediction mode, and determining, based on a size of thecurrent block, the secondary transform set applied to the current blockamong the plurality of secondary transform sets including the at leastone mixed secondary transform set when it is determined that the mixedsecondary transform set is used and determining, based on the size ofthe current block, the secondary transform set applied to the currentblock among residual secondary transform sets other than the at leastone mixed secondary transform set when it is determined that the mixedsecondary transform set is not used.

In another aspect of the present disclosure, provided is an apparatusfor decoding a video signal, including: a dequantization unit generatinga dequantized transform block by performing dequantization for a currentblock; a prediction mode acquiring unit acquiring an intra predictionmode of the current block; a secondary transform set determining unitdetermining a secondary transform set applied to the current block amonga plurality of secondary transform sets based on the intra predictionmode; a transform kernel deriving unit deriving a transform kernelapplied to the current block in the determined secondary transform set;and a secondary inverse transform unit performing secondary inversetransform for a top left specific area of the dequantized transformblock by using the derived transform kernel, in which the plurality ofsecondary transform sets may include at least one mixed secondarytransform set.

Preferably, the mixed secondary transform set may include at least one8×8 transform kernel applied to an 8×8 size area and at least one 4×4transform kernel applied to a 4×4 size area.

Preferably, when the plurality of secondary transform sets include aplurality of mixed secondary transform sets, the plurality of mixedsecondary transform sets may include different numbers of transformkernels, respectively.

Preferably, the transform kernel deriving unit may acquire a secondarytransform index indicating a transform kernel applied to the currentblock in the determined secondary transform set, and the secondarytransform index may be binarized by a truncated unary scheme based on amaximum number of available transform kernels in the determinedsecondary transform set.

Preferably, the secondary transform set determining unit may determinewhether to use the mixed secondary transform set based on the intraprediction mode, and determine, based on a size of the current block,the secondary transform set applied to the current block among theplurality of secondary transform sets including the at least one mixedsecondary transform set when it is determined that the mixed secondarytransform set is used and determines, based on the size of the currentblock, the secondary transform set applied to the current block amongresidual secondary transform sets other than the at least one mixedsecondary transform set when it is determined that the mixed secondarytransform set is not used.

Advantageous Effects

According to an embodiment of the present disclosure, a secondarytransform set is configured by considering various conditions in orderto apply secondary transform to efficiently select a transform used forthe secondary transform.

Effects obtainable in the present disclosure are not limited to theaforementioned effects and other unmentioned effects will be clearlyunderstood by those skilled in the art from the following description.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic block diagram of an encoder in which encoding of avideo signal is performed as an embodiment to which the presentdisclosure is applied.

FIG. 2 is a schematic block diagram of a decoder in which decoding of avideo signal is performed as an embodiment to which the presentdisclosure is applied.

FIGS. 3A-3D illustrate embodiments to which the present disclosure maybe applied, FIG. 3A is a diagram for describing block divisionstructures by QuadTree (QT) (hereinafter, referred to as ‘QT’), FIG. 3Bis a diagram for describing block division structures by Binary Tree(BT) (referred to as ‘TT’), FIG. 3C is a diagram for describing blockdivision structures by Ternary Tree (TT) (hereinafter, referred to as‘TT’), and FIG. 3D is a diagram for describing block division structuresby Asymmetric Tree (AT) (hereinafter, referred to as ‘AT’).

FIG. 4 is a schematic block diagram of transform and quantization units120 and 130 and dequantization and inverse transform units 140 and 150in an encoder as an embodiment to which the present disclosure isapplied.

FIG. 5 is a schematic block diagram of a dequantization unit and aninverse transform unit 220 and 230 in a decoder as an embodiment towhich the present disclosure is applied.

FIG. 6 is a table showing a transform configuration group to whichMultiple Transform Selection (MTS) is applied as an embodiment to whichthe present disclosure is applied.

FIG. 7 is a flowchart showing an encoding process in which MultipleTransform Selection (MTS) is performed as an embodiment to which thepresent disclosure is applied.

FIG. 8 is a flowchart showing a decoding process in which MultipleTransform Selection (MTS) is performed as an embodiment to which thepresent disclosure is applied.

FIG. 9 is a flowchart for describing a process of encoding an MTS flagand an MTS index as an embodiment to which the present disclosure isapplied.

FIG. 10 is a flowchart for describing a decoding process in whichhorizontal transform or vertical transform is applied to a row or acolumn based on an MTS flag and an MTS index as an embodiment to whichthe present disclosure is applied.

FIG. 11 is a flowchart of performing inverse transform based on atransform related parameter as an embodiment to which the presentdisclosure is applied.

FIG. 12 is a table showing allocation of a transform set for each intraprediction mode in an NSST as an embodiment to which the presentdisclosure is applied.

FIG. 13 is a calculation flow diagram for givens rotation as anembodiment to which the present disclosure is applied.

FIG. 14 illustrates one round configuration in 4×4 NSST constituted by agivens rotation layer and permutations as an embodiment to which thepresent disclosure is applied.

FIG. 15 is a block diagram for describing operations of forward reducedtransform and inverse reduced transform as an embodiment to which thepresent disclosure is applied.

FIG. 16 is a diagram illustrating a process of performing backward scanfrom 64^(th) to 17^(th) according to a backward scan order as anembodiment to which the present disclosure is applied.

FIG. 17 illustrates three forward scan orders for a transformcoefficient block (transform block) as an embodiment to which thepresent disclosure is applied.

FIG. 18 illustrates positions of valid transform coefficients and aforward scan order for each of 4×4 blocks when diagonal scan is appliedand 4×4 RST is applied in top-left 4×8 blocks as an embodiment to whichthe present disclosure is applied.

FIG. 19 illustrates a case where valid transform coefficients of two 4×4blocks are combined into one 4×4 block when diagonal scan is applied and4×4 RST is applied in top-left 4×8 blocks as an embodiment to which thepresent disclosure is applied.

FIG. 20 is a flowchart of encoding a video signal based on reducedsecondary transform as an embodiment to which the present disclosure isapplied.

FIG. 21 is a flowchart of decoding a video signal based on reducedsecondary transform as an embodiment to which the present disclosure isapplied.

FIG. 22 is a diagram illustrating a method for determining a transformtype applied to secondary transform according to an embodiment to whichthe present disclosure is applied.

FIG. 23 is a diagram illustrating a reduced transform structure based ona reduced factor as an embodiment to which the present disclosure may beapplied.

FIG. 24 is a diagram illustrating a method for performing decoding byadaptively applying reduced transform as an embodiment to which thepresent disclosure may be applied.

FIG. 25 is a diagram illustrating a method for performing decoding byadaptively applying reduced transform as an embodiment to which thepresent disclosure may be applied.

FIGS. 26 and 27 are diagrams illustrating examples of forward reducedsecondary transform and inverse reduced secondary transform and a pseudocode for deriving the same, as an embodiment to which the presentdisclosure is applied.

FIG. 28 is a diagram illustrating a method for applying reducedsecondary transform to a non-square region as an embodiment to which thepresent disclosure is applied.

FIG. 29 is a diagram illustrating reduced transform controlled by areduction factor as an embodiment to which the present disclosure isapplied.

FIG. 30 is a diagram illustrating an inverse transform method accordingto an embodiment to which the present disclosure is applied.

FIG. 31 is a diagram illustrating an inverse transform unit according toan embodiment to which the present disclosure is applied.

FIG. 32 illustrates a video coding system to which the presentdisclosure is applied.

FIG. 33 is a structural diagram of a content streaming system as anembodiment to which the present disclosure is applied.

MODE FOR INVENTION

Hereinafter, a configuration and an operation of an embodiment of thepresent disclosure will be described with reference to the accompanyingdrawings, the configuration and operation of the present disclosuredescribed by the drawings will be described as one embodiment, wherebythe technical spirit of the present disclosure and a core compositionand an operation thereof are not limited.

In addition, the term used in the present disclosure is selected as ageneral term widely used as possible now, in a specific case will bedescribed using terms arbitrarily selected by the applicant. In such acase, since the meaning is clearly stated in the detailed description ofthe part, it should not be interpreted simply by the name of the termused in the description of the present disclosure, and it should beunderstood that the meaning of the term should be interpreted.

In addition, terms used in the present disclosure may be replaced formore appropriate interpretation when there are general terms selected todescribe the invention or other terms having similar meanings. Forexample, signals, data, samples, pictures, frames, blocks, etc., may beappropriately replaced and interpreted in each coding process. Inaddition, partitioning, decomposition, splitting, and division may beappropriately replaced and interpreted in each coding process.

In this document, Multiple Transform Selection (MTS) may refer to amethod for performing transform using at least two transform types. Thismay also be expressed as an Adaptive Multiple Transform (AMT) orExplicit Multiple Transform (EMT), and likewise, mts_idx may also beexpressed as AMT_idx, EMT_idx, tu_mts_idx, AMT_TU_idx, EMT_TU_idx,transform index, or transform combination index and the presentdisclosure is not limited to the expressions.

FIG. 1 is a schematic block diagram of an encoder in which encoding of avideo signal is performed as an embodiment to which the presentdisclosure is applied.

Referring to FIG. 1, the encoder 100 may be configured to include animage division unit 110, a transform unit 120, a quantization unit 130,a dequantization unit 140, an inverse transform unit 150, a filteringunit 160, a decoded picture buffer (DPB) 170, an inter-prediction unit180, an intra-prediction unit 185, and an entropy encoding unit 190.

The image division unit 110 may divide an input image (or picture orframe) input into the encoder 100 into one or more processing units. Forexample, the processing unit may be a Coding Tree Unit (CTU), a CodingUnit (CU), a Prediction Unit (PU), or a Transform Unit (TU).

However, the terms are only used for the convenience of description ofthe present disclosure and the present disclosure is not limited to thedefinition of the terms. In addition, in the present disclosure, for theconvenience of the description, the term coding unit is used as a unitused in encoding or decoding a video signal, but the present disclosureis not limited thereto and may be appropriately interpreted according tothe present disclosure.

The encoder 100 subtracts a prediction signal (or a prediction block)output from the inter-prediction unit 180 or the intra-prediction unit185 from the input image signal to generate a residual signal (or aresidual block) and the generated residual signal is transmitted to thetransform unit 120.

The transform unit 120 may generate a transform coefficient by applyinga transform technique to the residual signal. A transform process may beapplied to a quadtree structure square block and a block (square orrectangle) divided by a binary tree structure, a ternary tree structure,or an asymmetric tree structure.

The transform unit 120 may perform a transform based on a plurality oftransforms (or transform combinations), and the transform scheme may bereferred to as multiple transform selection (MTS). The MTS may also bereferred to as an Adaptive Multiple Transform (AMT) or an EnhancedMultiple Transform (EMT).

The MTS (or AMT or EMT) may refer to a transform scheme performed basedon a transform (or transform combinations) adaptively selected from theplurality of transforms (or transform combinations).

The plurality of transforms (or transform combinations) may include thetransforms (or transform combinations) described in FIG. 6 of thepresent disclosure. In the present disclosure, the transform ortransform type may be expressed as, for example, DCT-Type 2, DCT-II,DCT2, or DCT-2.

The transform unit 120 may perform the following embodiments.

The present disclosure provides a method for designing an RST that maybe applied to a 4×4 block.

The present disclosure provides a configuration of a region to which the4×4 RST is to be applied, a method for arranging transform coefficientsgenerated after applying the 4×4 RST, a scan order of the arrangedtransform coefficients, a method for sorting and combining transformcoefficients generated for each block, and the like.

The present disclosure provides a method for coding a transform indexthat specifies the 4×4 RST.

The present disclosure provides a method for conditionally coding acorresponding transform index by checking whether a non-zero transformcoefficient exists in an unacceptable region when applying the 4×4 RST.

The present disclosure provides a method for conditionally coding thecorresponding transform index after coding a last non-zero transformcoefficient position, and then omitting relevant residual coding forpositions that are not accepted.

The present disclosure provides a method for applying differenttransform index coding and residual coding to a luma block and a chromablock when applying the 4×4 RST.

Detailed embodiments thereof will be described in more detail in thepresent disclosure.

The quantization unit 130 may quantize the transform coefficient andtransmits the quantized transform coefficient to the entropy encodingunit 190 and the entropy encoding unit 190 may entropy-code a quantizedsignal and output the entropy-coded quantized signal as a bitstream.

Although the converter 120 and the quantization unit 130 are describedas separate functional units, the present disclosure is not limitedthereto and may be combined into one functional unit. The dequantizationunit 140 and the inverse transform unit 150 may also be similarlycombined into one functional unit.

A quantized signal output from the quantization unit 130 may be used forgenerating the prediction signal. For example, inverse quantization andinverse transform are applied to the quantized signal through thedequantization unit 140 and the inverse transform unit 1850 in a loop toreconstruct the residual signal. The reconstructed residual signal isadded to the prediction signal output from the inter-prediction unit 180or the intra-prediction unit 185 to generate a reconstructed signal.

Meanwhile, deterioration in which a block boundary is shown may occurdue to a quantization error which occurs during such a compressionprocess. Such a phenomenon is referred to as blocking artifacts and thisis one of key elements for evaluating an image quality. A filteringprocess may be performed in order to reduce the deterioration. Blockingdeterioration is removed and an error for a current picture is reducedthrough the filtering process to enhance an image quality.

The filtering unit 160 applies filtering to the reconstructed signal andoutputs the applied reconstructed signal to a reproduction device ortransmits the output reconstructed signal to the decoded picture buffer170. The inter-prediction unit 170 may use the filtered signaltransmitted to the decoded picture buffer 180 as the reference picture.As such, the filtered picture is used as the reference picture in theinter-picture prediction mode to enhance the image quality and theencoding efficiency.

The decoded picture buffer 170 may store the filtered picture in orderto use the filtered picture as the reference picture in theinter-prediction unit 180.

The inter-prediction unit 180 performs a temporal prediction and/orspatial prediction in order to remove temporal redundancy and/or spatialredundancy by referring to the reconstructed picture. Here, since thereference picture used for prediction is a transformed signal that isquantized and inverse-quantized in units of the block at the time ofencoding/decoding in the previous time, blocking artifacts or ringingartifacts may exist.

Accordingly, the inter-prediction unit 180 may interpolate a signalbetween pixels in units of a sub-pixel by applying a low-pass filter inorder to solve performance degradation due to discontinuity orquantization of such a signal. Here, the sub-pixel means a virtual pixelgenerated by applying an interpolation filter and an integer pixel meansan actual pixel which exists in the reconstructed picture. As aninterpolation method, linear interpolation, bi-linear interpolation,wiener filter, and the like may be adopted.

An interpolation filter is applied to the reconstructed picture toenhance precision of prediction. For example, the inter-prediction unit180 applies the interpolation filter to the integer pixel to generate aninterpolated pixel and the prediction may be performed by using aninterpolated block constituted by the interpolated pixels as theprediction block.

Meanwhile, the intra-prediction unit 185 may predict the current blockby referring to samples in the vicinity of a block which is to besubjected to current encoding. The intra-prediction unit 185 may performthe following process in order to perform the intra prediction. First, areference sample may be prepared, which is required for generating theprediction signal. In addition, the prediction signal may be generatedby using the prepared reference sample. Thereafter, the prediction modeis encoded. In this case, the reference sample may be prepared throughreference sample padding and/or reference sample filtering. Since thereference sample is subjected to prediction and reconstructionprocesses, a quantization error may exist. Accordingly, a referencesample filtering process may be performed with respect to eachprediction mode used for the intra prediction in order to reduce such anerror.

The prediction signal generated through the inter-prediction unit 180 orthe intra-prediction unit 185 may be used for generating thereconstructed signal or used for generating the residual signal.

FIG. 2 is a schematic block diagram of a decoder in which decoding of avideo signal is performed as an embodiment to which the presentdisclosure is applied.

Referring to FIG. 2, the decoder 200 may be configured to include aparsing unit (not illustrated), an entropy decoding unit 210, adequantization unit 220, an inverse transform unit 230, a filtering unit240, a decoded picture buffer (DPB) unit 250, an inter-prediction unit260, and an intra-prediction unit 265.

In addition, a reconstructed video signal output through the decoder maybe reproduced through a reproduction device.

The decoder 200 may receive the signal output from the encoder 100 ofFIG. 1 and the received signal may be entropy-decoded through theentropy decoding unit 210.

The dequantization unit 220 acquires the transform coefficient from anentropy-decoded signal by using quantization step size information.

The inverse transform unit 230 inversely transforms the transformcoefficient to acquire the residual signal.

Here, the present disclosure provides a method for configuring atransform combination for each transform configuration group divided byat least one of a prediction mode, a block size or a block shape and theinverse transform unit 230 may perform inverse transform based on thetransform combination configured by the present disclosure. Further, theembodiments described in the present disclosure may be applied.

The inverse transform unit 230 may perform the following embodiments.

The present disclosure provides a method for reconstructing the videosignal based on reduced secondary transform.

The inverse transform unit 230 may derive a secondary transformcorresponding to a secondary transform index, performs inverse secondarytransform for the transform coefficient block by using the secondarytransform, and perform inverse primary transform for the block in whichthe inverse secondary transform is performed. Here, the secondarytransform refers to the reduced secondary transform and the reducedsecondary transform represents a transform in which N residual data (N×1residual vectors) are input to output L (L<N) transform coefficient data(L×1 transform coefficient vectors).

The present disclosure is characterized in that the reduced secondarytransform is applied to a specific region of the current block and thespecific region is a top-left M×M (M≤N) in the current block.

The present disclosure is characterized in that 4×4 reduced secondarytransform is applied to each of 4×4 blocks divided in the current blockswhen the inverse secondary transform is performed.

The present disclosure is characterized in that it is determined whetherthe secondary transform index is obtained based on the position of thelast non-zero transform coefficient in the transform coefficient block.

The present disclosure is characterized in that when the last non-zerotransform coefficient is not positioned in the specific region, thesecondary transform index is obtained and the specific region indicatesremaining regions other than a position when the non-zero transformcoefficient may exist when the transform coefficients are arrangedaccording to the scan order in the case where the reduced secondarytransform is applied.

The inverse transform unit 230 may derive a transform combinationcorresponding to a primary transform index and perform an inverseprimary transform by using the transform combination. Here, the primarytransform index corresponds to any one of a plurality of transformcombinations constituted by a combination of DST7 and/or DCT8 and thetransform combination includes a horizontal transform and a verticaltransform. In this case, the horizontal transformation and the verticaltransformation correspond to either the DST7 or the DCT8.

Although the dequantization unit 220 and the inverse transform unit 230are described as separate functional units, the present disclosure isnot limited thereto and may be combined into one functional unit.

The obtained residual signal is added to the prediction signal outputfrom the inter-prediction unit 260 or the intra-prediction unit 265 togenerate the reconstructed signal.

The filtering unit 240 applies filtering to the reconstructed signal andoutputs the applied reconstructed signal to a generation device ortransmits the output reconstructed signal to the decoded picture bufferunit 250. The inter-prediction unit 250 may use the filtered signaltransmitted to the decoded picture buffer unit 260 as the referencepicture.

In the present disclosure, the embodiments described in the transformunit 120 and the respective functional units of the encoder 100 may beequally applied to the inverse transform unit 230 and the correspondingfunctional units of the decoder, respectively.

FIG. 3 illustrates embodiments to which the present disclosure may beapplied, FIG. 3A is a diagram for describing block division structuresby QuadTree (QT) (hereinafter, referred to as ‘QT’), FIG. 3B is adiagram for describing block division structures by Binary Tree (BT)(referred to as ‘TT’), FIG. 3C is a diagram for describing blockdivision structures by Ternary Tree (TT) (hereinafter, referred to as‘TT’), and FIG. 3D is a diagram for describing block division structuresby Asymmetric Tree (AT) (hereinafter, referred to as ‘AT’).

In video coding, one block may be divided based on a QuadTree (QT). Inaddition, one sub block divided by the QT may be further dividedrecursively using the QT. A leaf block which is not QT-divided anylonger may be divided by at least one scheme of Binary Tree (BT),Ternary Tree (TT), and Asymmetric Tree (AT). The BT may have two typesof divisions: horizontal BT (2N×N, 2N×N) and vertical BT (N×2N, N×2N).The TT may have two types of divisions: horizontal TT (2N×½N, 2N×N,2N×½N) and vertical TT (½N×2N, N×2N, ½N×2N). The AT may have four typesof divisions: horizontal-up AT (2N×½N, 2N×3/2N), horizontal-down AT(2N×3/2N, 2N×½N), vertical-left AT (½N×2N, 3/2N×2N), vertical-right AT(3/2N×2N, ½N×2N). Each of the BT, the TT, and the AT may be furtherdivided recursively by using the BT, the TT, and the AT.

FIG. 3A illustrates an example of QT division. Block A may be dividedinto four sub blocks A0, A1, A2, and A3 by the QT. Sub block A1 may bedivided into four sub blocks B0, B1, B2, and B3 by the QT again.

FIG. 3B illustrates an example of BT division. Block B3 which is notdivided by the QT any longer may be divided into vertical BT (C0, C1) orhorizontal BT (D0, D1). Like block C0, each sub block may be furtherrecursively divided like a form of horizontal BT (E0, E1) or vertical BT(F0, F1).

FIG. 3C illustrates an example of TT division. Block B3 which is notdivided by the QT any longer may be divided into vertical TT (C0, C1,C2) or horizontal TT (D0, D1, D2). Like block C1, each sub block may befurther recursively divided like a form of horizontal TT (E0, E1, E2) orvertical TT (F0, F1, F2).

FIG. 3D illustrates an example of AT division. Block B3 which is notdivided by the QT any longer may be divided into vertical AT (C0, C1) orhorizontal AT (D0, D1). Like block C1, each sub block may be furtherrecursively divided like a form of horizontal AT (E0, E1) or vertical TT(F0, F1).

Meanwhile, the BT, TT, and AT divisions may be used together and made.For example, the sub block divided by the BT may be divided by the TT orAT. Further, the sub block divided by the TT may be divided by the BT orAT. Further, the sub block divided by the AT may be divided by the BT orTT. For example, after the horizontal BT division, each sub block may bedivided into vertical BTs or after the vertical BT division, each subblock may be divided into horizontal BTs. The two types of divisionmethods are different from each other in terms of a division order, butare the same as each other in terms of a final division shape.

Further, when the block is divided, an order of searching for the blockmay be variously defined. In general, searching may be performed fromleft to right and from top to bottom, and the searching for the blockmay mean an order of deciding whether to further divide each dividedsub-block, mean a coding order of each sub block when each sub block isno longer divided, or mean a search order when referring to informationof another neighboring block in the sub block.

FIGS. 4 and 5 illustrate embodiments to which the present disclosure isapplied, FIG. 4 is a schematic block diagram of transform andquantization units 120 and 130 and dequantization and inverse transformunits 140 and 150 in an encoder, and FIG. 5 is a schematic block diagramof dequantization and inverse transform units 220 and 230 in a decoder.

Referring to FIG. 4, the transform and quantization units 120 and 130may include a primary transform unit 121, a secondary transform unit122, and a quantization unit 130. The dequantization and inversetransform units 140 and 150 may include a dequantization unit 140, aninverse secondary transform unit 151, and an inverse primary transformunit 152.

Referring to FIG. 5, the dequantization and inverse transform units 220and 230 may include a dequantization unit 220, an inverse secondarytransform unit 231, and an inverse primary transform unit 232.

In the present disclosure when the transform is performed, the transformmay be performed through a plurality of steps. For example, two steps ofthe primary transform and the secondary transform may be applied asillustrated in FIG. 4 or more transform steps may be used according toan algorithm. Here, the primary transform may also be referred to as acore transform.

The primary transform unit 121 may apply the primary transform to theresidual signal and here, the primary transform may be defined in atable in the encoder and/or the decoder.

The primary transform may adopt Discrete Cosine Transform type 2(hereinafter, referred to as ‘DCT2’).

Alternatively, only in a specific case, Discrete Sine Transform-type 7(hereinafter, referred to as ‘DST7’) may be adopted. For example, theDST7 may be applied to the 4×4 block in the intra-prediction mode.

Further, the primary transform may adopt combinations of varioustransforms DST 7, DCT 8, DST 1, and DCT 5 of the multiple transformselection (MTS). For example, FIG. 6 may be adopted.

The secondary transform unit 122 may apply the secondary transform to aprimary transformed signal and here, the secondary transform may bedefined in the table in the encoder and/or the decoder.

As an embodiment, the secondary transform may conditionally adopt anon-separable secondary transform (hereinafter, referred to as ‘NSST’).For example, the NSST may be applied only to the intra-prediction blockand may have a transform set applicable to each prediction mode group.

Here, the prediction mode group may be configured based on symmetry withrespect to a prediction direction. For example, since prediction mode 52and prediction mode 16 are symmetrical based on prediction mode 34(diagonal direction), the same transform set may be applied by formingone group. In this case, when the transform for prediction mode 52 isapplied, input data is transposed and then applied because predictionmode 52 has the same transform set as prediction mode 16.

Meanwhile, since the symmetry for the direction does not exist in thecase of a planar mode and a DC mode, each mode has a different transformset and the corresponding transform set may be constituted by twotransforms. In respect to the remaining direction modes, each transformset may be constituted by three transforms.

As another embodiment, the secondary transform may adopt combinations ofvarious transforms DST 7, DCT 8, DST 1, and DCT 5 of the multipletransform selection (MTS). For example, FIG. 6 may be adopted.

As another embodiment, the DST 7 may be applied to the secondarytransform.

As another embodiment, the secondary transform may not be applied to theentire primary transformed block but may be applied only to a top-leftspecific region. For example, when the block size is 8×8 or more, 8×8NSST is applied and when the block size is less than 8×8, a 4×4secondary transform may be applied. In this case, the block may bedivided into 4×4 blocks and then the 4×4 secondary transform may beapplied to each divided block.

As another embodiment, even in the case of 4×N/N×4 (N>=16), the 4×4secondary transform may be applied.

The secondary transform (e.g., NSST), the 4×4 secondary transform, andan 8×8 secondary transform will be described in more detail withreference to FIGS. 12 to 15 and other embodiments in the specification.

The quantization unit 130 may perform quantization for the secondarytransformed signal.

The dequantization and inverse transform units 140 and 150 perform theabove-described process in reverse, and a redundant description thereofwill be omitted.

FIG. 5 is a schematic block diagram of a dequantization unit 220 and aninverse transform unit 230 in a decoder.

Referring to FIG. 5, the dequantization and inverse transform units 220and 230 may include a dequantization unit 220, an inverse secondarytransform unit 231, and an inverse primary transform unit 232.

The dequantization unit 220 acquires the transform coefficient from anentropy-decoded signal by using quantization step size information.

The inverse secondary transform unit 231 performs an inverse secondarytransform for the transform coefficients. Here, the inverse secondarytransform represents an inverse transform of the secondary transformdescribed in FIG. 4.

As another embodiment, the secondary transform may adopt combinations ofvarious transforms DST 7, DCT 8, DST 1, and DCT 5 of the multipletransform selection (MTS). For example, FIG. 6 may be adopted.

The inverse primary transform unit 232 performs an inverse primarytransform for the inverse secondary transformed signal (or block) andobtains the residual signal. Here, the inverse primary transformrepresents the inverse transform of the primary transform described inFIG. 4.

As an embodiment, the primary transform may adopt combinations ofvarious transforms DST 7, DCT 8, DST 1, and DCT 5 of the multipletransform selection (MTS). For example, FIG. 6 may be adopted.

As an embodiment of the present disclosure, the DST 7 may be applied tothe primary transform.

As an embodiment of the present disclosure, the DST 8 may be applied tothe primary transform.

The present disclosure provides a method for configuring a transformcombination for each transform configuration group divided by at leastone of a prediction mode, a block size or a block shape and the inverseprimary transform unit 232 may perform the inverse transform based onthe transform combination configured by the present disclosure. Further,the embodiments described in the present disclosure may be applied.

FIG. 6 is a table showing a transform configuration group to whichMultiple Transform Selection (MTS) is applied as an embodiment to whichthe present disclosure is applied.

Transform Configuration Group to which Multiple Transform Selection(MTS) is Applied

In the present disclosure, a j-th transform combination candidate fortransform configuration group G_(i) is represented by a pair shown inEquation 1 below.(H(G _(i) ,j),V(G _(i) ,j))  [Equation 1]

Here, H(G_(i), j) indicates the horizontal transform for the j-thcandidate, and V(G_(i), j) indicates the vertical transform for the j-thcandidate. For example, in FIG. 6, H(G₃, 2)=DST7, V(G₃, 2)=DCT8 may berepresented. Depending on a context, a value assigned to H(G_(i), j) orV(G_(i), j) may be a nominal value to distinguish transformations, as inthe example above or may be an index value indicating the transform ormay be a 2 dimensional (D) matrix for the transform.

Further, in the present disclosure, a 2D matrix value for DCT and DSTmay be represented as shown in Equation 2 and 3 below.DCT type 2: C _(N) ^(II),DCT type 8: C _(N) ^(VIII)  [Equation 2]DST type 7: S _(N) ^(VII),DST type 4: S _(N) ^(IV)  [Equation 3]

Here, whether the DST or the DCT is represented by S or C and a typenumber is expressed in superscripts in the form of Roman numerals and Nin a subscript indicates an N×N transform. Further, 2D matrices such asthe C_(N) ^(II) and the S_(N) ^(IV) suppose that column vectors form atransform basis.

Referring to FIG. 6, the transform configuration groups may bedetermined based on the prediction mode and the number of groups may bea total of six groups G0 to G5. In addition, G0 to G4 correspond to acase where intra prediction is applied, and G5 represents transformcombinations (or transform sets and transform combination sets) appliedto the residual block generated by the inter prediction.

One transform combination may be constituted by a horizontal transform(or row transform) applied to rows of a corresponding 2D block and avertical transform (or column transform) applied to columns.

Here, each of all of the transform configuration groups may have fourtransform combination candidates. The four transform combinations may beselected or determined through transform combination indexes of 0 to 3and transmitted by encoding the transform combination index from theencoder to the decoder.

As an embodiment, the residual data (or residual signal) obtainedthrough the intra prediction may have different statisticalcharacteristics according to the intra prediction mode. Therefore, asillustrated in FIG. 6, transforms other than a general cosine transformmay be applied to each intra prediction mode.

Referring to FIG. 6, a case of using 35 intra prediction modes and acase of using 67 intra prediction modes are illustrated. A plurality oftransform combinations may be applied to each transform configurationgroup divided in each intra prediction mode column. For example, theplurality of transform combinations may be constituted by four (rowdirection transforms and column direction transforms) combinations. As aspecific example, DST-7 and DST-5 may be applied in a row (horizontal)direction and a column (vertical) direction in group 0, and as a result,a total of four combinations are available.

Since a total of transform kernel combinations may be applied to eachintra prediction mode, a transform combination index for selecting oneof the transform kernel combinations may be transmitted every transformunit. In the present disclosure, the transform combination index may becalled MTS index and expressed as mts_idx.

Further, in addition to the transform kernels presented in FIG. 6 above,a case where DCT2 is optimal for both the row direction and the columndirection due to characteristics of the residual signal may occur.Accordingly, the MTS flag is defined for each coding unit to adaptivelyperform the transform. Here, when the MTS flag is 0, DCT2 may be appliedto both the row direction and the column direction and when the MTS flagis 1, one of four combinations may be selected or determined through theMTS index.

As an embodiment, when the MTS flag is 1, if the number of non-zerotransform coefficients for one transform unit is not greater than athreshold, the DST-7 may be applied both the row direction and thecolumn direction is not applied without applying the transform kernelsof FIG. 6. For example, the threshold may be set to 2, which may be setdifferently based on the block size or the size of the transform unit.This is also applicable to other embodiments in the specification.

As an embodiment, if the number of non-zero transform coefficients isnot greater than the threshold, by first parsing the transformcoefficient values, the amount of additional information transmissionmay be reduced by applying the DST-7 without parsing the MTS index.

As an embodiment, when the MTS flag is 1, if the number of non-zerotransform coefficients is greater than the threshold for one transformunit, the MTS index may be parsed and the horizontal transform and thevertical transform may be determined based on the MTS index.

As an embodiment, the MTS may be applied only when both a width and aheight of the transform unit is equal to or smaller than 32.

As an embodiment, FIG. 6 may be preconfigured through off-line training.

As an embodiment, the MTS index may be defined as one index which maysimultaneously indicate the horizontal transform and the verticaltransform. Alternatively, the MTS index may be separately defined as ahorizontal transform index and a vertical transform index.

As an embodiment, the MTS flag or the MTS index may be defined at atleast one level of a sequence, a picture, a slice, a block, a codingunit, a transform unit, or a prediction unit. For example, the MTS flagor the MTS index may be defined at at least one level of a sequenceparameter set (SPS), the coding unit, or the transform unit. Further, asan example, a syntax flag for enabling/disabling the MTS may be definedat at least one level of the sequence parameter set (SPS), a pictureparameter set (PPS), or a slice header.

As another embodiment, the transform combination (horizontal transformor vertical transform) corresponding to the transform index may beconfigured without dependence on the MTS flag, the prediction mode,and/or a block shape. For example, the transform combination may beconfigured by at least one of DCT2, DST7, and/or DCT8. As a specificexample, when the transform index is 0, 1, 2, 3, or 4, each transformcombination may be (DCT2, DCT2), (DST7, DST7), (DCT8, DST7), (DST7,DCT8), or (DCT8, DCT8).

FIG. 7 is a flowchart showing an encoding process in which MultipleTransform Selection (MTS) is performed as an embodiment to which thepresent disclosure is applied.

In the present disclosure, an embodiment in which transforms are aseparately applied to the horizontal direction and the verticaldirection is basically described, but the transform combination may beconfigured as non-separable transforms.

Alternatively, the transform combination may be configured by a mixtureof separable transforms and non-separable transforms. In this case, whenthe non-separable transform is used, row/column transform selection orhorizontal/vertical direction selection may not be required and onlywhen the separable transform is selected, the transform combinations ofFIG. 6 may be used.

Further, schemes proposed by the present disclosure may be appliedregardless of the primary transform or the secondary transform. That is,there is no limit that the schemes should be applied only to any one ofboth the primary transform and the secondary transform and the schemesmay be applied to both the primary transform and the secondarytransform. Here, the primary transform may mean a transform fortransforming the residual block first and the secondary transform maymean a transform for applying the transform to the block generated as aresult of the primary transform.

First, the encoder may determine the transform configuration groupcorresponding to the current block. Here, the transform configurationgroup may mean the transform configuration group of FIG. 6 and thepresent disclosure is not limited thereto and the transformconfiguration group may be constituted by other transform combinations.

The encoder may perform a transform for candidate transform combinationsusable in the transform configuration group (S720).

As a result of performing the transform, the encoder may determine orselect a transform combination having a smallest rate distortion (RD)cost (S730).

The encoder may encode the transform combination index corresponding tothe selected transform combination (S740).

FIG. 8 is a flowchart showing a decoding process in which MultipleTransform Selection (MTS) is performed as an embodiment to which thepresent disclosure is applied.

First, the decoder may determine the transform configuration group forthe current block (S810).

The decoder may parse (or acquire) the transform combination index fromthe video signal and here, the transform combination index maycorrespond to any one of the plurality of transform combinations in thetransform configuration group (S820). For example, the transformconfiguration group may include Discrete Sine Transform type (DST) 7 andDiscrete Cosine Transform type (DST) 8. The transform combination indexmay be referred to as the MTS index.

As an embodiment, the transform configuration group may be configuredbased on at least one of the prediction mode, the block size, or theblock shape of the current block.

The decoder may derive the transform combination corresponding to thetransform combination index (S830). Here, the transform combination maybe constituted by the horizontal transform and the vertical transformand may include at least one of the DST-7 or the DCT-8.

Further, the transform combination may mean the transform combinationdescribed in FIG. 6, but the present disclosure is not limited thereto.That is, the transform combination may be configured by other transformcombinations depending on other embodiments in the present disclosure.

The decoder may perform the inverse transform for the current blockbased on the transform combination (S840). When the transformcombination is constituted by the row (horizontal) transform and thecolumn (vertical) transform, the column (vertical) transform may beapplied after applying the row (horizontal) transform first. However,the present disclosure is not limited thereto and the transform ordermay be reversed or when the transform combination is constituted by thenon-separable transforms, the non-separable transform may be immediatelyapplied.

As an embodiment, when the vertical transform or the horizontaltransform is the DST-7 or the DCT-8, the inverse transform of the DST-7or the inverse transform of the DCT-8 may be applied to each row andthen applied to each row.

As an embodiment, in respect to the vertical transform or the horizontaltransform, different transform may be applied to each row and/or eachcolumn.

As an embodiment, the transform combination index may be acquired basedon the MST flag indicating whether the MTS is performed. That is, thetransform combination index may be obtained when the MTS is performedaccording to the MTS flag.

As an embodiment, the decoder may check whether the number of non-zerotransform coefficients is greater than the threshold. In this case, thetransform may be obtained when the number of non-zero transformcoefficients is greater than the threshold.

As an embodiment, the MTS flag or the MTS index may be defined at leastone level of a sequence, a picture, a slice, a block, a coding unit, atransform unit, or a prediction unit.

As an embodiment, the inverse transform may be applied only when boththe width and the height of the transform unit is equal to or smallerthan 32.

On the other hand, as another embodiment, a process of determining thetransform configuration group and a process of parsing the transformcombination index may be performed at the same time. Alternatively, stepS810 may be preconfigured and omitted in the encoder and/or the decoder.

FIG. 9 is a flowchart for describing a process of encoding an MTS flagand an MTS index as an embodiment to which the present disclosure isapplied.

The encoder may determine whether the Multiple Transform Selection (MTS)is applied to the current block (S910).

When the Multiple Transform Selection (MTS) is applied, the encoder mayencode MTS flag=1 (S920).

In addition, the encoder may determine the MTS index based on at leastone of the prediction mode, the horizontal transform, and the verticaltransform of the current block (S930). Here, the MTS index may mean anindex indicating any one of the plurality of transform combinations foreach intra prediction mode and the MTS index may be transmitted for eachtransform unit.

When the MTS index is determined, the encoder may encode the MTS index(S940).

On the other hand, when the Multiple Transform Selection (MTS) is notapplied, the encoder may encode MTS flag=0 (S920).

FIG. 10 is a flowchart for describing a decoding process in whichhorizontal transform or vertical transform is applied to a row or acolumn based on an MTS flag and an MTS index as an embodiment to whichthe present disclosure is applied.

The decoder may parse the MTS flag from the bitstream (S1010). Here, theMTS flag may indicate whether the Multiple Transform Selection (MTS) isapplied to the current block.

The decoder may determine whether the Multiple Transform Selection (MTS)is applied to the current block based on the MTS flag (S1020). Forexample, it may be checked whether the MTS flag is 1.

When the MTS flag is 1, the decoder may check whether the number ofnon-zero transform coefficients is greater than (or equal to or greaterthan) the threshold (S1030). For example, the threshold may be set to 2,which may be set differently based on the block size or the size of thetransform unit.

When the number of non-zero transform coefficients is greater than thethreshold, the decoder may parse the MTS index (S1040). Here, the MTSindex may mean any one of the plurality of transform combinations foreach intra prediction mode or inter prediction mode and the MTS indexmay be transmitted for each transform unit. Alternatively, the MTS indexmay mean an index indicating any one transform combination defined in apreconfigured transform combination table and here, the preconfiguredtransform combination table may mean FIG. 6, but the present disclosureis limited thereto.

The decoder may derive or determine the horizontal transform and thevertical transform based on at least one of the MTS index and theprediction mode (S1050).

Alternatively, the decoder may derive the transform combinationcorresponding to the MTS index. For example, the decoder may derive ordetermine the horizontal transform and the vertical transformcorresponding to the MTS index.

When the number of non-zero transform coefficients is not greater thanthe threshold, the decoder may apply a preconfigured vertical inversetransform (S1060). For example, the vertical inverse transform may bethe inverse transform of the DST7.

In addition, the decoder may apply a preconfigured horizontal inversetransformation for each row (S1070). For example, the horizontal inversetransform may be the inverse transform of the DST7. That is, when thenumber of non-zero transform coefficients is not greater than thethreshold, a transform kernel preconfigured by the encoder or decodermay be used. For example, a transform kernel (e.g., DCT-2, DST-7, orDCT-8) that is not defined in the transform combination tableillustrated in FIG. 6, but is widely used may be used.

Meanwhile, when the MTS flag is 0, the decoder may apply thepreconfigured vertical inverse transform for each column (S1080). Forexample, the vertical inverse transform may be the inverse transform ofthe DCT2.

In addition, the decoder may apply the preconfigured horizontal inversetransformation for each row (S1090). For example, the horizontal inversetransform may be the inverse transform of the DCT2. That is, when theMTS flag is 0, the transform kernel preconfigured by the encoder ordecoder may be used. For example, the transform kernel that is notdefined in the transform combination table illustrated in FIG. 6, but iswidely used may be used.

FIG. 11 is a flowchart of performing inverse transform based on atransform related parameter as an embodiment to which the presentdisclosure is applied.

The decoder to which the present disclosure is applied may obtainsps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1110). Here,sps_mts_intra_enabled_flag indicates whether tu_mts_flag exists in aresidual coding syntax of an intra coding unit. For example, whensps_mts_intra_enabled_flag=0, tu_mts_flag does not exist in the residualcoding syntax of the intra coding unit and whensps_mts_intra_enabled_flag=0, tu_mts_flag exists in the residual codingsyntax of the intra coding unit. In addition, sps_mts_inter_enabled_flagindicates whether tu_mts_flag exists in the residual coding syntax ofthe inter coding unit. For example, when sps_mts_inter_enabled_flag=0,tu_mts_flag does not exist in the residual coding syntax of the intracoding unit and when sps_mts_inter_enabled_flag=0, tu_mts_flag exists inthe residual coding syntax of the inter coding unit.

The decoder may obtain tu_mts_flag based on sps_mts_intra_enabled_flagor sps_mts_inter_enabled_flag (S1120). For example, whensps_mts_intra_enabled_flag=1 or sps_mts_inter_enabled_flag=1, thedecoder may obtain tu_mts_flag. Here, tu_mts_flag indicates whethermultiple transform selection (hereinafter, referred to as “MTS”) isapplied to a residual sample of a luma transform block. For example,when tu_mts_flag=0, the MTS is not applied to the residual sample of theluma transform block and when tu_mts_flag=1, the MTS is applied to theresidual sample of the luma transform block.

As another example, at least one of the embodiments of the presentdocument may be applied to the tu_mts_flag.

The decoder may obtain mts_idx based on tu_mts_flag (S1130). Forexample, when tu_mts_flag=1, the decoder may obtain mts_idx. Here,mts_idx indicates which transform kernel is applied to luma residualsamples along the horizontal and/or vertical direction of a currenttransform block.

For example, at least one of the embodiments of the present document maybe applied to mts_idx. As a specific example, at least one of theembodiments of FIG. 6 may be applied.

The decoder may derive the transform kernel corresponding to mts_idx(S1140). For example, the transform kernel corresponding to the mts_idxmay be defined by being divided into the horizontal transform and thevertical transform.

As another example, different transform kernels may be applied to thehorizontal transform and the vertical transform. However, the presentdisclosure is not limited thereto, and the same transform kernel may beapplied to the horizontal transform and the vertical transform.

As an embodiment, mts_idx may be defined as shown in Table 1 below.

TABLE 1 mts_idx[ x0 ][ y0 ] trTypeHor trTypeVer 0 0 0 1 1 1 2 2 1 3 1 24 2 2

In addition, the decoder may perform the inverse transform based on thetransform kernel (S1150).

In FIG. 11 above, an embodiment is primarily described in whichtu_mts_flag is obtained to determine whether to apply MTS and mts_idx isobtained according to a tu_mts_flag value which is obtained later todetermine the transform kernel, but the present disclosure is notlimited thereto. As an example, the decoder parses mts_idx directlywithout parsing tu_mts_flag to determine the transform kernel. In thiscase, Table 1 described above may be used. That is, when the mts_idxvalue indicates 0, DCT-2 may be applied in the horizontal/verticaldirection and when the mts_idx value indicates a value other than 0,DST-7 and/or DCT-8 may be applied according to the mts_idx value.

As another embodiment of the present disclosure, a decoding process ofperforming the transform process is described.

The decoder may check a transform size nTbS (S10). Here, the transformsize nTbS may be a variable representing a horizontal sample size ofscaled transform coefficients.

The decoder may check a transform kernel type trType (S20). Here, thetransform kernel type trType may be a variable representing the type oftransform kernel and various embodiments of the present document may beapplied. The transform kernel type trType may include a horizontaltransform kernel type trTypeHor and a vertical transform kernel typetrTypeVer.

Referring to Table 1, when the transform kernel type trType is 0, thetransform kernel type may represent DCT2, when the transform kernel typetrType is 1, the transform kernel type may represent DST7, and when thetransform kernel type trType is 2, the transform kernel type mayrepresent DCT8.

The decoder may perform a transform matrix multiplication based on atleast one of the transform size nTbS or the transform kernel type (S30).

As another example, when the transform kernel type is 1 and thetransform size is 4, a predetermined transform matrix 1 may be appliedwhen performing the transform matrix multiplication.

As another example, when the transform kernel type is 1 and thetransform size is 8, a predetermined transform matrix 2 may be appliedwhen performing the transform matrix multiplication.

As another example, when the transform kernel type is 1 and thetransform size is 16, a predetermined transform matrix 3 may be appliedwhen performing the transform matrix multiplication.

As another example, when the transform kernel type is 1 and thetransform size is 32, a predefined transform matrix 4 may be appliedwhen performing the transform matrix multiplication.

Similarly, when the transform kernel type is 2 and the transform size is4, 8, 16, or 32, predefined transform matrices 5, 6, 7, and 8 may beapplied, respectively.

Here, each of the predefined transform matrices 1 to 8 may correspond toany one of various types of transform matrices. As an example, thetransform matrix of the type illustrated in FIG. 6 may be applied.

The decoder may derive a transform sample (or transform coefficient)based on transform matrix multiplication (S40).

Each of the above embodiments may be used, but the present disclosure isnot limited thereto, and may be used in combination with the aboveembodiments and other embodiments of the present disclosure.

FIG. 12 is a table showing allocation of a transform set for each intraprediction mode in an NSST as an embodiment to which the presentdisclosure is applied.

Non-Separable Secondary Transform (NSST)

The secondary transform unit may apply the secondary transform to aprimary transformed signal and here, the secondary transform may bedefined in the table in the encoder and/or the decoder.

As an embodiment, the secondary transform may conditionally adopt anon-separable secondary transform (hereinafter, referred to as ‘NSST’).For example, the NSST may be applied only to the intra-prediction blockand may have a transform set applicable to each prediction mode group.

Here, the prediction mode group may be configured based on symmetry withrespect to a prediction direction. For example, since prediction mode 52and prediction mode 16 are symmetrical based on prediction mode 34(diagonal direction), the same transform set may be applied by formingone group. In this case, when the transform for prediction mode 52 isapplied, input data is transposed and then applied because predictionmode 52 has the same transform set as prediction mode 16.

Meanwhile, since the symmetry for the direction does not exist in thecase of a planar mode and a DC mode, each mode has a different transformset and the corresponding transform set may be constituted by twotransforms. In respect to the remaining direction modes, each transformset may be constituted by three transforms. However, the presentdisclosure is not limited thereto, and each transform set may beconstituted by a plurality of transforms.

In an embodiment, a transform set table other than that illustrated inFIG. 12 may be defined. For example, as shown in Table 2 below, thetransform set may be determined from a predefined table according to anintra prediction mode (or an intra prediction mode group). A syntaxindicating a specific transform in the transform set determinedaccording to the intra prediction mode may be signaled from the encoderto the decoder.

TABLE 2 IntraPredMode Tr. set index   IntraPredMode < 0 1 0 < =IntraPredMode < = 1 0  2 < = IntraPredMode < = 12 1 13 < = IntraPredMode< = 23 2 24 < = IntraPredMode < = 44 3 45 < = IntraPredMode < = 55 2 56< = IntraPredMode    1

Referring to Table 2, a predefined transform set may be allocated to thegrouped intra prediction modes (or intra prediction mode groups). Here,the IntraPredMode value may be a mode value transformed in considerationof Wide Angle Intra Prediction (WAIP).

FIG. 13 is a calculation flow diagram for givens rotation as anembodiment to which the present disclosure is applied.

As another embodiment, the NSST may not be applied to the entire primarytransformed block but may be applied only to a top-left 8×8 region. Forexample, when the block size is 8×8 or more, 8×8 NSST is applied andwhen the block size is less than 8×8, 4×4 NSST is applied and in thiscase, the block is divided into 4×4 blocks and then, the 4×4 NSST isapplied to each of the divided blocks.

As another embodiment, even in the case of 4×N/N×4 (N>=16), the 4×4 NSSTmay be applied.

Since both the 8×8 NSST and the 4×4 NSST follow a transformationcombination configuration described in this document and are thenon-separable transforms, the 8×8 NSST receives 64 data and outputs 64data and the 4×4 NSST has 16 inputs and 16 outputs.

Both the 8×8 NSST and the 4×4 NSST are configured by a hierarchicalcombination of Givens rotations. A matrix corresponding to one Givensrotation is shown in Equation 4 below and a matrix product is shown inEquation 5 below.

$\begin{matrix}{R_{\theta} = \begin{bmatrix}{\cos\;\theta} & {{- {s{in}}}\;\theta} \\{\sin\;\theta} & {\cos\;\theta}\end{bmatrix}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack \\{{l_{m} = {{x_{m}\cos\;\theta} - {x_{n}\sin\;\theta}}}{l_{n} = {{x_{m}\sin\;\theta} + {x_{n}\cos\;\theta}}}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

As illustrated in FIG. 13, since one Givens rotation rotates two data,in order to process 64 data (for the 8×8 NSST) or 16 data (for the 4×4NSST), a total of 32 or 8 Givens rotations are required.

Therefore, a bundle of 32 or 8 is used to form a Givens rotation layer.Output data for one Givens rotation layer is transferred as input datafor a next Givens rotation layer through a determined permutation.

FIG. 14 illustrates one round configuration in 4×4 NSST constituted by agivens rotation layer and permutations as an embodiment to which thepresent disclosure is applied.

Referring to FIG. 14, it is illustrated that four Givens rotation layersare sequentially processed in the case of the 4×4 NSST. As illustratedin FIG. 14, the output data for one Givens rotation layer is transferredas the input data for the next Givens rotation layer through adetermined permutation (i.e., shuffling).

As illustrated in FIG. 14, patterns to be permutated are regularlydetermined and in the case of the 4×4 NSST, four Givens rotation layersand the corresponding permutations are combined to form one round.

In the case of the 8×8 NSST, six Givens rotation layers and thecorresponding permutations form one round. The 4×4 NSST goes through tworounds and the 8×8 NSST goes through four rounds. Different rounds usethe same permutation pattern, but applied Givens rotation angles aredifferent. Accordingly, angle data for all Givens rotations constitutingeach transform need to be stored.

As a last step, one permutation is further finally performed on the dataoutput through the Givens rotation layers, and corresponding permutationinformation is stored separately for each transform. In forward NSST,the corresponding permutation is performed last and in inverse NSST, acorresponding inverse permutation is applied first on the contrarythereto.

In the case of the inverse NSST, the Givens rotation layers and thepermutations applied to the forward NSST are performed in the reverseorder and rotation is performed by taking a negative value even for anangle of each Givens rotation.

FIG. 15 is a block diagram for describing operations of forward reducedtransform and inverse reduced transform as an embodiment to which thepresent disclosure is applied.

Reduced Secondary Transform (RST)

When it is assumed that an orthogonal matrix representing one transformhas an N×N form, a reduced transform (hereinafter, referred to as ‘RT’)leaves only R transform basis vectors among N transform basis vectors(R<N). A matrix for forward RT generating the transform coefficients isgiven by Equation 6 below.

$\begin{matrix}{T_{R \times N} = \begin{bmatrix}t_{11} & t_{12} & t_{13} & \ldots & t_{1N} \\t_{21} & t_{22} & t_{23} & \; & t_{2N} \\\; & \vdots & \; & \ddots & \vdots \\t_{R\; 1} & t_{R\; 2} & t_{R\; 3} & \ldots & t_{RN}\end{bmatrix}} & \left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack\end{matrix}$

Since a matrix for an inverse RT becomes a transpose matrix of theforward RT matrix, the application of the forward RT and the inverse RTis illustrated as illustrated in FIG. 15. Here, a reduction factor isdefined as R/N (R<N).

The number of elements of the reduced transform is R * N, which issmaller than the size of the entire matrix (N * N). In other words, arequired matrix is R/N of the entire matrix. Further, the number ofrequired multiplications is R×N and is lower than the original N×N byR/N. When the reduced transform is applied, R coefficients are provided,and as a result, only R coefficient values may be transmitted instead ofN coefficients.

Assuming a case of applying the RT to the top-left 8×8 block of thetransform block which goes through the primary transform is assumed, theRT may be referred to as an 8×8 reduced secondary transform (8×8 RST).

When the R value of Equation 6 above is 16, the forward 8×8 RST has a16×64 matrix form and the inverse 8×8 RST has a 64×16 matrix form.

Further, the transform set configuration which is the same as thatillustrated in FIG. 12 may be applied even to the 8×8 RST. That is, acorresponding 8×8 RST may be applied according to the transform set inFIG. 12.

As an embodiment, when one transform set is constituted by two or threetransforms according to the intra prediction mode in FIG. 12, one of amaximum of 4 transforms including a case of not applying the secondarytransform may be configured to be selected. Here, one transform may beregarded as an identity matrix.

When indexes of 0, 1, 2, and 3 are assigned to the four transforms,respectively, a syntax element called an NSST index may be signaled foreach transform block, thereby designating a corresponding transform.That is, in the case of the NSST, the 8×8 NSST may be designated for the8×8 top-left block through the NSST index and the 8×8 RST may bedesignated in an RST configuration. Further, in this case, index 0 maybe allocated to a case where the identity matrix, i.e., the secondarytransform is not applied.

When the forward 8×8 RST shown in Equation 6 is applied, 16 validtransform coefficients are generated, and as a result, it may beregarded that 64 input data constituting an 8×8 region are reduced to 16output data. From the perspective of a two-dimensional region, only aone-quarter region is filled with the valid transform coefficient.Accordingly, a 4×4 top-left region in FIG. 16 may be filled with 16output data obtained by applying the forward 8×8 RST.

FIG. 16 is a diagram illustrating a process of performing backward scanfrom 64^(th) to 17^(th) according to a backward scan order as anembodiment to which the present disclosure is applied.

FIG. 16 illustrates scanning from the 17^(th) coefficient to the 64^(th)coefficient when the forward scanning order starts from 1 (in theforward scanning order). However, FIG. 16 illustrates the backward scanand this illustrates performing the backward scanning from the 64^(th)coefficient to the 17^(th) coefficient.

Referring to FIG. 16, the top-left 4×4 region is a region of interest(ROI) to which the valid transform coefficient is allocated and theremaining region is empty. That is, a value of 0 may be allocated to theremaining region by default.

If there is a valid transform coefficient other than 0 in a region otherthan the ROI region of FIG. 16, this means that the 8×8 RST is notapplied, and as a result, in this case, NSST index coding correspondingthereto may be omitted.

Conversely, if there is no non-zero transform coefficient outside of theROI region of FIG. 16 (if the 8×8 RST is applied, when 0 is allocated tothe region other than the ROI), there is a possibility that the 8×8 RSTwill be applied, and as a result, the NSST index may be coded.

As such, conditional NSST index coding may be performed after theresidual coding process because it is necessary to check the existenceof the non-zero transform coefficient.

The present disclosure provides a method for designing an RST andassociated optimization methods which may be applied to the 4×4 blockfrom an RST structure. The embodiments disclosed in the presentdisclosure may be applied to the 8×8 RST or another type of transform inaddition to the 4×4 RST.

FIG. 17 illustrates three forward scan orders for a transformcoefficient block (transform block) as an embodiment to which thepresent disclosure is applied.

Embodiment 1: RST Applicable to 4×4 Block

A non-separable transform that may be applied to one 4×4 block is a16×16 transform. That is, when data elements constituting the 4×4 blockare arranged in a row-first or column-first order, a 16×1 vector is usedto apply the non-separable transform.

The forward 16×16 transform is constituted by 16 row-wise transformedbasis vectors and when an inner product is applied to the 16×1 vectorand each transform basis vector, the transform coefficient for thetransform basis vector is obtained. A process of obtaining transformcoefficients corresponding to all of 16 transform basis vectors isequivalent to multiplying the 16×16 non-separable transform matrix bythe input 16×1 vector.

The transform coefficients obtained by the matrix product have a 16×1vector form, and statistical characteristics may be different for eachtransform coefficient. For example, when a 16×1 transform coefficientvector is constituted by a 0^(th) element to a 15^(th) element, avariance of the 0^(th) element may be greater than the variance of the15^(th) element. In other words, as the element is positioned former, acorresponding variance value of the element is larger, so that theelement may have a larger energy value.

When the inverse 16×16 non-separable transform is applied from the 16×1transform coefficient, an original 4×4 block signal may be restored.When the forward 16×16 non-separable transform is an orthonormaltransform, the corresponding inverse 16×16 transform may be obtainedthrough the transpose matrix for the forward 16×16 transform.

When the 16×1 transform coefficient vector is multiplied by the inverse16×16 non-separable transform matrix, data in the form of the 16×1vector may be obtained and when the obtained data are arranged in therow-first or column-first order which is first applied, the 4×4 blocksignal may be restored.

As described above, elements constituting the 16×1 transform coefficientvector may have different statistical characteristics.

If transform coefficients arranged at a former side (close to an 0^(th)element) have larger energy, a signal may be restored, which is quiteclose to the original signal even though the inverse transform isapplied to some transform coefficients which first appear without usingall transform coefficients. For example, when the inverse 16×16non-separable transform is constituted by 16 column basis vectors, onlyL column basis vectors are left to form a 16×L matrix. In addition, whena 16×L matrix and an L×1 are multiplied by each other after only Limportant transform coefficients among the transform coefficients (L×1vector), the 16×1 vector may be restored, which has a small error fromoriginal input 16×1 vector data.

As a result, since only L coefficients are used for data restoring, theL×1 transform coefficient vector is obtained instead of the 16×1transform coefficient vector when obtaining the transform coefficient.That is, when an L×16 transform is configured by selecting Lcorresponding row-wise vectors in the forward 16×16 non-separabletransform matrix and the configured L×16 transform is multiplied by the16×1 input vector, L important transform coefficients may be obtained.

The L value has a range of 1≤L≤16 and in general, L vectors may beselected by an arbitrary method among 16 transform basis vectors, but itmay be advantageous in terms of coding efficiency to select transformbasis vectors having a high importance in terms of energy of the signalfrom the viewpoint of coding and decoding.

Embodiment 2: Configuration of Application Region of 4×4 RST andArrangement of Transform Coefficients

The 4×4 RST may be applied as the secondary transform, and may beapplied secondarily to a block to which a primary transform such asDCT-type 2 is applied. When the size of the block to which the primarytransform is applied is N×N, the size of the block to which the primarytransform is applied is generally larger than 4×4. Therefore, whenapplying the 4×4 RST to the N×N block, there may two methods as follows.

Embodiment 2-1) The 4×4 RST is not applied to all N×N regions, but maybe applied only to some regions. For example, the 4×4 RST may be appliedonly to the top-left M×M region (M≤N).

Embodiment 2-2) A region to which the secondary transform is to beapplied may be divided into 4×4 blocks and then the 4×4 RST may beapplied to each divided block.

As an embodiment, embodiments 2-1) and 2-2) may be mixed and applied.For example, only the top-left M×M region may be divided into 4×4 blocksand then the 4×4 RST may be applied.

As an embodiment, the secondary transform may be applied only to thetop-left 8×8 region and when N×N block is equal to or larger than 8×8,the 8×8 RST may be applied and when the N×N block is smaller than 8×8(4×4, 8×4, and 4×8), the N×N block may be divided into 4×4 blocks andthen the 4×4 RST may be applied to each of 4×4 blocks as in embodiment2-2). Further, even in the case of 4×N/N×4 (N>=16), the 4×4 NSST may beapplied.

When L (1≤L<16) transform coefficients are generated after applying the4×4 RST, a degree of freedom for how the L transform coefficients arearranged is generated. However, since there will be a predeterminedorder when processing the transform coefficient in a residual codingstep, coding performance may vary depending on how the L transformcoefficients are arranged in a 2D block.

For example, in the case of residual coding of HEVC, coding starts froma position farthest from a DC position. This is to enhance the codingperformance by taking advantage of a fact that a quantized coefficientvalue is zero or close to zero as moving away from the DC position.

Therefore, it may be advantageous in terms of the coding performance toarrange more important coefficients with high energy for the L transformcoefficients so that the L transform coefficients are coded later in theorder of residual coding.

FIG. 17 illustrates three forward scan orders in units of a 4×4transform block (coefficient group (CG) applied in HEVC. The residualcoding follows the reverse order of the scan order of FIG. 17 (i.e.,coding is performed in the order of 16 to 1).

Since three scan orders presented in FIG. 17 are selected according tothe intra prediction mode, the present disclosure may be configured todetermine the scan order according to the intra prediction modesimilarly even for the L transform coefficients.

FIG. 18 illustrates positions of valid transform coefficients and aforward scan order for each of 4×4 blocks when diagonal scan is appliedand 4×4 RST is applied in top-left 4×8 blocks as an embodiment to whichthe present disclosure is applied.

When following a diagonal scan order in FIG. 17 and dividing thetop-left 4×8 block into 4×4 blocks and applying the 4×4 RST to each 4×4block, if the L value is 8 (i.e., if only 8 transform coefficients among16 transform coefficients are left), the transform coefficients may bepositioned as in FIG. 18.

Only half of respective 4×4 blocks may have the transform coefficientsand a value of 0 may be applied to positions marked with X by default.

Accordingly, the residual coding is performed by arranging L transformcoefficients for each 4×4 block according to the scan order illustratedin FIG. 17 and assuming that 16−L remaining positions of each 4×4 blockare filled with zeros.

FIG. 19 illustrates a case where valid transform coefficients of two 4×4blocks are combined into one 4×4 block when diagonal scan is applied and4×4 RST is applied in top-left 4×8 blocks as an embodiment to which thepresent disclosure is applied.

Referring to FIG. 19, L transform coefficients arranged in two 4×4blocks may be combined into one. In particular, when the L value is 8,since the transform coefficients of two 4×4 blocks are combined whilecompletely filling one 4×4 block, no transform coefficient is also leftin the other one 4×4 block.

Accordingly, since most residual coding is not required with respect tothe empty 4×4 block, corresponding coded_sub_block_flag may be codedwith 0.

Further, as an embodiment of the present disclosure, various schemes maybe applied even to how transform coefficients of two 4×4 blocks aremixed. The transform coefficients may be combined according to a randomorder, but the present disclosure may provide the following methods.

1) The transform coefficients two 4×4 blocks are alternately mixed inthe scan order. That is, in FIG. 18, when the transform coefficients fora top block are C₀ ^(u), C₁ ^(u), C₂ ^(u), C₃ ^(u), C₄ ^(u), C₅ ^(u), C₆^(u), and C₇ ^(u) and the transform coefficients for a bottom block areC₀ ^(l), C₁ ^(l), C₂ ^(l), C₃ ^(l), C₄ ^(l), C₅ ^(l), C₆ ^(l), and C₇^(l) the transform coefficients may be alternately mixed one by one likeC₀ ^(u), C₀ ^(l), C₁ ^(u), C₁ ^(l), C₂ ^(u), C₂ ^(l), . . . , C₇ ^(u)and C₇ ^(l). Alternatively, the order of C_(#) ^(u) and C_(#) ^(l) maybe changed. In other words, the order may be configured so that C_(#)^(l) appears first.

2) The transform coefficients for a first 4×4 block may be firstarranged and then the transform coefficients for a second 4×4 block maybe arranged. In other words, the transform coefficients may be connectedand arranged like C₀ ^(u), C₁ ^(u), . . . , C₇ ^(u), C₀ ^(l), C₁ ^(l), .. . , and C₇ ^(l)˜. Alternatively, the order may be changed like C₀^(l), C₁ ^(l), . . . , C₇ ^(l), C₀ ^(u), C₁ ^(u), . . . , and C₇ ^(u).

Embodiment 3: Method for Coding NSST Index for 4×4 RST

When the 4×4 RST is applied as illustrated in FIG. 18, L+1-th to 16-thmay be filled with the 0 value according to the transform coefficientscan order for each 4×4 blocks.

Accordingly, when a non-zero value is generated at L+1-th to 16-thpositions even in one of two 4×4 blocks, it may be known that this caseis a case where the 4×4 RST is not applied.

When the 4×4 RST also has a structure in which one of the transform setsprepared as the NSST is selected and applied, a transform index (whichmay be referred to as an NSST index in the embodiment) for whichtransform to apply may be signaled.

It is assumed that any decoder may know the NSST index through bitstreamparsing and the parsing is performed after residual decoding.

When the residual decoding is performed and it is confirmed that atleast one non-zero transform coefficient exists between L+1-th to 16-th,the 4×4 RST is not applied, and thus the NSST index may be configured tonot be parsed.

Accordingly, the NSST index is selectively parsed only when necessary toreduce signaling cost.

If the 4×4 RST is applied to a plurality of 4×4 blocks in a specificregion as illustrated in FIG. 18 (for example, the same 4×4 RST may beapplied to all of the plurality of 4×4 blocks or different 4×4 RSTs maybe applied), the 4×4 RST applied to the all 4×4 blocks may be designatedthrough one NSST index. In this case, the same 4×4 RST may be designatedor the 4×4 RST applied to each of all 4×4 blocks may be designated.

Since whether the 4×4 RST is applied to the all 4×4 blocks by one NSSTindex, it may be checked whether non-zero transform coefficients existat L+1-th to 16-th positions for the all 4×4 blocks during a residualdecoding process. As a checking result, when the non-zero transformcoefficient exists at a position (L+1-th to 16-th positions) which isnot accepted even in one 4×4 block, the NSST index may be configured tonot be coded.

The NSST index may be signaled separately for the luma block and thechroma block, and in the case of the chroma block, separate NSST indexesmay be signaled for Cb and Cr, and one NSST index may be shared.

When one NSST index is shared for Cb and Cr, 4×4 RST designated by thesame NSST index may be applied. In this case, the 4×4 RSTs for Cb and Crmay be the same or the NSST index may be the same, but separate 4×4 RSTsmay be provided.

To apply the conditional signaling to the shared NSST index, it ischecked whether non-zero transform coefficients exist at L+1-th to 16-thfor all 4×4 blocks for Cb and Cr and when the non-zero transformcoefficient exists, the NSST index may be configured to not be signaled.

As illustrated in FIG. 19, even for a case where the transformcoefficients for two 4×4 blocks are combined, it is checked whether thenon-zero transform coefficient exists at a position where the validtransform coefficient does not exist when the 4×4 RST is applied andthen it may be determined whether the NSST is signaled.

For example, as illustrated in FIG. 19(b), when the L value is 8 and thevalid transform coefficients do not exist for one 4×4 blocks at the timeof applying the 4×4 RST (a block marked with X), coded_sub_block_flag ofa block where the valid transform coefficients do not exist may bechecked. In this case, when coded_sub_block_flag is 1, the NSST indexmay be configured to not be signaled.

Embodiment 4: Optimization Method for Case where Coding for NSST Indexis Performed Before Residual Coding

When coding for the NSST index is performed before residual coding,whether to apply the 4×4 RST is predetermined, and as a result, residualcoding may be omitted for positions whether 0 is allocated to thetransform coefficient.

Here, whether to apply the 4×4 RST may be configured to be known throughthe NSST index. For example, when the NSST index is 0, the 4×4 RST isnot applied.

Alternatively, the NSST index may be signaled through a separate syntaxelement (e.g., NSST flag). For example, if a separate syntax element iscalled the NSST flag, the NSST flag is parsed first to determine whetherthe 4×4 RST is applied, and if the NSST flag value is 1, the residualcoding may be omitted for positions where no valid transform coefficientmay exist.

As an embodiment, when the residual coding is performed, a last non-zerotransform coefficient position on the TU is coded first. When the codingfor the NSST index is performed after coding the last non-zero transformcoefficient position and it is assumed that the 4×4 RST is applied tothe position of the last non-zero transform coefficient, if the lastnon-zero transform coefficient position is determined as a positionwhere the non-zero transform coefficient may not be generated, the 4×4RST may be configured not to be applied to the last non-zero transformcoefficient position without coding the NSST index.

For example, since in the case of positions marked with X in FIG. 18,valid transformation coefficients are not positioned when the 4×4 RST isapplied (e.g., the positions may be filled with zero values), when thelast non-zero transform coefficient is positioned in the region markedwith X, the coding for the NSST index may be omitted. When the lastnon-zero transform coefficient is not positioned in the region markedwith X, the coding of the NSST index may be performed.

As an embodiment, when it is checked whether to apply the 4×4 RST byconditionally coding the NSST index after the coding for the lastnon-zero transform coefficient position, the remaining residual codingportion may be processed by two following schemes.

1) In case of not applying the 4×4 RST, general residual coding is keptas it is. That is, the coding is performed under the assumption that thenon-zero transform coefficient may exist at any position from thenon-zero transform coefficient position to DC.

2) When the 4×4 RST is applied, since no transform coefficient existsfor a specific position or a specific 4×4 block (e.g., an X position ofFIG. 18, which may be filled with 0 by default), the residual for thecorresponding position or block may not be performed.

For example, in a case of reaching the position marked with X in FIG.18, coding for sig_coeff_flag may be omitted. Here, sig_coeff_flag meansa flag for whether the non-zero transform coefficient exists at acorresponding position.

When transform coefficients of two blocks are combined as illustrated inFIG. 19, the coding for coded_sub_block_flag may be omitted for the 4×4blocks allocated to 0 and a corresponding value may be derived to 0 andall corresponding 4×4 blocks may be derived to zero values withoutseparate coding.

In a case where the NSST index is coded after coding the non-zerotransform coefficient position, when an x position P_(x) and a yposition P_(y) of the last non-zero transform coefficient are smallerthan T_(x) and T_(y), respectively, the NSST index coding may be omittedand the 4×4 RST may not be applied.

For example, a case of T_(x)=1 and T_(y)=1 means that the NSST indexcoding is omitted for a case where the non-zero transform coefficientexists at the DC position.

A scheme of determining whether to code the NSST index throughcomparison with the threshold may be differently applied to luma andchroma. For example, different T_(x) and T_(y) may be applied to theluma and the chroma and the threshold may be applied to the luma and notapplied to the chroma. Or vice versa.

Two methods described above, that is, a first method for omitting theNSST index coding when the non-zero transform coefficient is located ina region where the valid transform coefficient does not exist and asecond method for omitting the NSST index coding when each of an Xcoordinate and a Y coordinate for the non-zero transform coefficient issmaller than a predetermined threshold may be simultaneously applied.

For example, a threshold for a position coordinate of the last non-zerotransform coefficient may be first checked and then it may be checkedwhether the last non-zero transform coefficient is located in the regionwhere the valid transform coefficient does not exist. Alternatively, theorder may be changed.

Methods presented in Embodiment 4 may be applied even to the 8×8 RST.That is, when the last non-zero transform coefficient is located in aregion other than the top-left 4×4 in the top-left 8×8 region, thecoding for the NSST index may be omitted and if not, the NSST indexcoding may be performed.

Further, when both X and Y coordinate values for the non-zero transformcoefficient are less than a threshold, the coding for the NSST index maybe omitted. Alternatively, two methods may be applied together.

Embodiment 5: Applying Different NSST Index Coding and Residual CodingSchemes to Luma and Chroma when Applying RST

The schemes described in Embodiments 3 and 4 may be differently appliedto luma and chroma, respectively. That is, the NSST index coding andresidual coding schemes for the luma and the chroma may be differentlyapplied.

For example, the luma may adopt the scheme of Embodiment 4 and thechroma may adopt the scheme of Embodiment 3. Alternatively, the luma mayadopt the conditional NSST index coding presented in Embodiment 3 or 4and the chroma may not adopt the conditional NSST index coding. Or viceversa.

FIG. 20 is a flowchart of encoding a video signal based on reducedsecondary transform as an embodiment to which the present disclosure isapplied.

The encoder may determine (or select) the forward secondary transformbased on at least one of the prediction mode, the block shape, and/orthe block size of the current block (S2010). In this case, a candidateof the forward secondary transform may include at least one of theembodiments of FIG. 6 and/or FIG. 12.

The encoder may determine an optimal forward secondary transform throughRate Distortion optimization. The optimal forward secondary transformmay correspond to one of a plurality of transform combinations and theplurality of transform combinations may be defined by a transform index.For example, for the RD optimization, results of performing all of theforward secondary transform, quantization, residual coding, etc., may becompared for respective candidates. In this case, an equation such ascost=rate+λ·distortion or cost=distortion+λ·rate may be used, but thepresent disclosure is not limited thereto.

The encoder may signal a secondary transform index corresponding to theoptimal forward secondary transform (S2020). Here, the secondarytransform index may adopt other embodiments described in the presentdisclosure.

For example, the secondary transform index may adopt the transform setconfiguration of FIG. 12. Since one transform set is constituted by twoor three transforms according to the intra prediction mode, one of amaximum of four transforms may be configured to be selected in additionto a case of not applying the secondary transform. When indexes of 0, 1,2, and 3 are assigned to the four transforms, respectively, an appliedtransform may be designated by signaling the secondary transform indexfor each transform coefficient block. In this case, index 0 may beallocated to a case where the identity matrix, i.e., the secondarytransform is not applied.

As another embodiment, the signaling of the secondary transform indexmay be performed in any one step of 1) before residual coding, 2) in themiddle of residual coding (after coding the non-zero transformcoefficient position), or 3) after residual coding. The embodiments willbe described below in detail.

1) Method for Signaling Secondary Transform Index Before Residual Coding

The encoder may determine the forward secondary transform.

The encoder may signal the secondary transform index corresponding tothe forward secondary transform.

The encoder may code the position of the last non-zero transformcoefficient.

The encoder may perform residual coding for syntax elements other thanthe position of the last non-zero transform coefficient.

2) Method for Signaling Secondary Transform Index in Middle of ResidualCoding

The encoder may determine the forward secondary transform.

The encoder may code the position of the last non-zero transformcoefficient.

When the non-zero transform coefficient is not located in a specificregion, the encoder may code the secondary transform index correspondingto the forward secondary transform. Here, in the case where the reducedsecondary transform is applied, the specific region represents aremaining region other than the position where the non-zero transformcoefficient may exist when the transform coefficients are arrangedaccording to the scan order. However, the present disclosure is notlimited thereto.

The encoder may perform residual coding for syntax elements other thanthe position of the last non-zero transform coefficient.

3) Method for Signaling Secondary Transform Index Before Residual Coding

The encoder may determine the forward secondary transform.

The encoder may code the position of the last non-zero transformcoefficient.

When the non-zero transform coefficient is not located in a specificregion, the encoder may perform residual coding for syntax elementsother than the position of the last non-zero transform coefficient.Here, in the case where the reduced secondary transform is applied, thespecific region represents a remaining region other than the positionwhere the non-zero transform coefficient may exist when the transformcoefficients are arranged according to the scan order. However, thepresent disclosure is not limited thereto.

The encoder may code the secondary transform index corresponding to theforward secondary transform.

Meanwhile, the encoder may perform the forward first order transform forthe current block (residual block) (S2030). Here, step S2010 and/or stepS2020 may be similarly applied to the forward primary transform.

The encoder may perform the forward secondary transform for the currentblock by using the optimal forward secondary transform (S2040). Forexample, the optimal forward secondary transform may be the reducedsecondary transform. The reduced secondary transform refers to atransform in which N residual data (N×1 residual vectors) are input andL (L<N) transform coefficient data (L×1 transform coefficient vectors)are output.

As an embodiment, the reduced secondary transform may be applied to aspecific region of the current block. For example, when the currentblock is N×N, the specific region may mean atop-left N/2×N/2 region.However, the present disclosure is not limited thereto and may bedifferently configured according to at least one of the prediction mode,the block shape, or the block size. For example, when the current blockis N×N, the specific region may mean a top-left M×M region (M≤N).

Meanwhile, the encoder performs quantization for the current block togenerate a transform coefficient block (S2050).

The encoder performs entropy encoding for the transform coefficientblock to generate the bitstream.

FIG. 21 is a flowchart of decoding a video signal based on reducedsecondary transform as an embodiment to which the present disclosure isapplied.

The decoder may obtain the secondary transform index from the bitstream(S2110). Here, the secondary transform index may adopt other embodimentsdescribed in the present disclosure. For example, the secondarytransform index may include at least one of the embodiments of FIG. 6and/or FIG. 12.

As another embodiment, the obtaining of the secondary transform indexmay be performed in any one step of 1) before residual coding, 2) in themiddle of residual coding (after decoding the non-zero transformcoefficient position), or 3) after residual coding.

The decoder may derive the secondary transform corresponding to thesecondary transform index (S2120). In this case, the candidate of theforward secondary transform may include at least one of the embodimentsof FIG. 6 and/or FIG. 12.

However, steps S2110 and S2120 are embodiments and the presentdisclosure is not limited thereto. For example, the decoder may notobtain the secondary transform index, but derive the secondary transformbased on at least one of the prediction mode, the block shape, and/orthe block size of the current block.

Meanwhile, the decoder may obtain the transform coefficient block byentropy-decoding the bitstream and perform dequantization for thetransform coefficient block (S2130).

The decoder may perform the inverse secondary transform for thedequantized transform coefficient block (S2140). For example, theinverse secondary transform may be the reduced secondary transform. Thereduced secondary transform refers to a transform in which N residualdata (N×1 residual vectors) are input and L (L<N) transform coefficientdata (L×1 transform coefficient vectors) are output.

As an embodiment, the reduced secondary transform may be applied to aspecific region of the current block. For example, when the currentblock is N×N, the specific region may mean atop-left N/2×N/2 region.However, the present disclosure is not limited thereto and may bedifferently configured according to at least one of the prediction mode,the block shape, or the block size. For example, when the current blockis N×N, the specific region may mean a top-left M×M region (M≤N) orM×L(M≤N, L≤N).

In addition, the decoder may perform the inverse primary transform forthe inverse secondary transform result (S2150).

The decoder generates the residual block through step S2150 and theresidual block and the prediction block are added to generate areconstruction block.

Embodiment 6: Method for Configuring Mixed Secondary Transform Set

In an embodiment, proposed is a method for configuring a secondarytransform set for considering various conditions in applying secondarytransform.

In the present disclosure, the secondary transform indicates transformperformed with respect to all or some primary transformed coefficientsafter applying primary transform at the encoder side as described aboveand the secondary transform may be referred to as Non SeparableSecondary Transform (NSST), Low-Frequency Non-Separable Transform(LFNST), etc. The decoder may apply the primary transform to all or somedequantized transform coefficients after applying the secondarytransform to all or some dequantized transform coefficients.

In addition, in the present disclosure, the mixed secondary transformset represents a transform set applicable to the secondary transform,and the present disclosure is not limited to such a name. For example,the mixed secondary transform (or transform kernel or transform type)set may be referred to as a secondary transform group, a secondarytransform table, a secondary transform candidate, a secondary transformcandidate list, a mixed secondary transform group, a mixed secondarytransform table, a mixed secondary transform candidate, a mixedsecondary transform candidate list, etc., and the mixed secondarytransform set may include a plurality of transform kernels (or transformtypes).

The secondary transform may be applied to a subblock having a specificsize on a top left side according to a predefined condition in thecurrent block. In the conventional video compression technique, a 4×4secondary transform set or an 8×8 secondary transform set is usedaccording to the size of a selected subblock. In this case, the 4×4secondary transform set includes only a transform kernel (hereinafter,referred to as a 4×4 transform kernel) applied to a 4×4 size region (orblock), and the 8×8 secondary transform set includes only a transformkernel (hereinafter, referred to as an 8×8 transform kernel) applied toan 8×8 size region (or block). In other words, in the conventional videocompression technique, the secondary transform set is constituted onlyby a transform kernel having a limited size according to the size of theregion to which the secondary transform is applied.

Accordingly, the present disclosure proposes a mixed secondary transformset including transform kernels that may be applied to regions havingvarious sizes.

As an embodiment, the size (i.e., the size of the region to which thecorresponding transform kernel is applied) of the transform kernelincluded in the mixed secondary transform set is not fixed, but may bevariably determined (or set). For example, the mixed secondary transformset may include the 4×4 transform kernel and the 8×8 transform kernel.

In addition, in an embodiment, the number of transform kernels includedin the mixed secondary transform set is not fixed, but may be variablydetermined (or set). In other words, the mixed secondary transform setmay include a plurality of transform sets, and each transform set mayinclude a different number of transform kernels. For example, a firsttransform set may include three transform kernels, and a secondtransform set may include four transform kernels.

In addition, in an embodiment, an order (or priority) between thetransform kernels included in the mixed secondary transform set is notfixed, but may be variably determined (or set). In other words, themixed secondary transform set may include a plurality of transform sets,and the order between the transform kernels in each transform set may beindividually defined. In addition, different indices may be mapped (orallocated) between the transform kernels within each transform set. Forexample, assuming that the first transform set and the second transformset each include a first transform kernel, a second transform kernel,and a third transform kernel, index values of 1, 2, and 3 may be mappedto the first transform kernel, the second transform kernel, and thethird transform kernel, respectively, within the first transform set andindex values of 3, 2, and 1 may be mapped to the first transform kernel,the second transform kernel, and the third transform kernel,respectively, within the second transform set.

Hereinafter, a method for determining the priority (or order) betweenthe transform kernels in the mixed secondary transform set will bedescribed in detail.

When the secondary transform is applied, the decoder may determine (orselect) the secondary transform set applied to the current processingblock according to a predefined condition. The secondary transform setmay include the mixed secondary transform set according to an embodimentof the present disclosure. In addition, the decoder may determine atransform kernel applied to the secondary transform of the currentprocessing block within the determined secondary transform set using thesecondary transform index signaled from the encoder. The secondarytransform index may indicate the transform kernel applied to thesecondary transform of the current processing block within thedetermined secondary transform set. In one embodiment, the secondarytransform index may be referred to as an NSST index, an LFNST index,etc.

As described above, since the secondary transform index is signaled fromthe encoder to the decoder, it is reasonable in terms of compressionefficiency to allocate a lower index to a transform kernel that occursrelatively more frequently to encode/decode with fewer bits.Accordingly, various embodiments of configuring the secondary transformset in consideration of priority will be described below.

In an embodiment, the priority between the transform kernels in thesecondary transform set may be determined differently according to thesize of the region (or subblock) to which the secondary transform isapplied. For example, when the size of the current processing block isequal to or larger than a predetermined size, the 8×8 transform kernelmay be used more frequently, and as a result, the encoder/decoder mayallocate relatively fewer transform indices to the 8×8 transform kernelthan the 4×4 transform kernel. As an example, the mixed secondarytransform set may be configured as in Table 3 below.

TABLE 3 NSST index 4 × 4 NSST Set 8 × 8 NSST Set Mixed NSST Set 1 4 × 41st Kernel  8 × 8 1st Kernel  8 × 8 1st Kernel  2 4 × 4 2nd Kernel 8 × 82nd Kernel 8 × 8 2nd Kernel 3 4 × 4 3rd Kernel 8 × 8 3rd Kernel 4 × 41st Kernel  . . . . . . . . . . . .

In Table 3, the embodiment is described by assuming that the NSST isused as the secondary transform, but the present disclosure is notlimited to such a name. As described above, the secondary transformaccording to an embodiment of the present disclosure may be referred toas Non Separable Secondary Transform (NSST), Low-Frequency Non-SeparableTransform (LFNST), etc. Referring to Table 3, the secondary transformset of the conventional video compression technique is constituted onlyby transform kernels of the size of the region to which the secondarytransform is applied (i.e., 4×4 NSST Set and 8×8 NSST Set in Table 3).The mixed secondary transform set (i.e., mixed NSST set) according to anembodiment of the present disclosure may include the 8×8 transformkernel and the 4×4 transform kernel.

In Table 3, it is assumed that the size of the current processing blockis equal to or larger than the predetermined size. That is, when aminimum value of the width or height of the current block is equal to orlarger than a predefined value (e.g., 8), the secondary transform set asillustrated in Table 3 may be used. The mixed secondary transform setmay include the 8×8 transform kernel and the 4×4 transform kernel andthe 8×8 transform kernel is highly likely to be used, and as a result, alow index value may be allocated to the 8×8 transform kernel.

In addition, in an embodiment, the priority between the transformkernels in the secondary transform set may be determined based on theorder (1st, 2nd, and 3rd) of the secondary transform kernels. Forexample, a first 4×4 secondary transform kernel may have a higherpriority than a second 4×4 secondary transform kernel and a lower indexvalue may be allocated to the first 4×4 secondary transform kernel. Asan example, the mixed secondary transform set may be configured as inTable 4 below.

TABLE 4 Mixed NSST Set Mixed NSST Set Mixed NSST Set NSST index Type 1Type 2 Type 3 1 8 × 8 3rd Kernel  8 × 8 1st Kernel  4 × 4 1st Kernel  28 × 8 2nd Kernel 8 × 8 2nd Kernel 8 × 8 1st Kernel  3 8 × 8 1st Kernel 4 × 4 1st Kernel  4 × 4 2nd Kernel 4 N.A 4 × 4 2nd Kernel 8 × 8 2ndKernel 5 N.A 4 × 4 3rd Kernel . . . . . .

In Table 4, the embodiment is described by assuming that the NSST isused as the secondary transform, but the present disclosure is notlimited to such a name. As described above, the secondary transformaccording to an embodiment of the present disclosure may be referred toas NSST, LFNST, etc.

Referring to Table 4, the mixed secondary transform set (i.e., mixedNSST set types 1, 2, and 3) according to an embodiment of the presentdisclosure may include the 8×8 transform kernel and/or the 4×4 transformkernel. In other words, like Mixed NSST Set Types 2 and 3, the mixedsecondary transform set may include the 8×8 transform kernel and/or the4×4 transform kernel, and the priority between the 8×8 transform kernelsand the priority between the 4×4 transform kernels may be configuredbased on the order of respective secondary transform kernels.

Embodiment 7: Method for Configuring Mixed Secondary Transform Set

In an embodiment of the present disclosure, proposed is a method fordetermining the secondary transform set in consideration of variousconditions. Specifically, a method for determining the secondarytransform set based on the intra prediction mode and/or the size of theblock is proposed.

The encoder/decoder may configure the transform set suitable for thesecondary transform of the current block based on the intra predictionmode. As an embodiment, the proposed method may be applied together withEmbodiment 6 described above. For example, the encoder/decoder mayconfigure the secondary transform set based on the intra prediction modeand perform the secondary transform using transform kernels of varioussizes included in each secondary transform set.

In an embodiment, the secondary transform set may be determinedaccording to the intra prediction mode based on Table 5 below.

TABLE 5 Intra 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Mode 17 18 19 2021 22 23 24 25 26 27 28 29 30 31 32 33 Mixed 1 1 1 1 1 0 0 0 0 0 0 0 0 00 0 1 Type 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 Intra 34 35 36 37 38 39 4041 42 43 44 45 46 47 48 49 50 Mode 51 52 53 54 55 56 57 58 59 60 61 6263 64 65 66 67 Mixed 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 Type 1 1 0 0 0 00 0 0 0 0 0 0 0 0 1 0

Referring to Table 5, the encoder/decoder may determine whether to apply(or configure) the mixed secondary transform set described in Embodiment6 above based on the intra prediction mode. When the mixed secondarytransform set is not applied, the encoder/decoder may apply (orconfigure) the secondary transform set described in FIGS. 12 to 14above.

Specifically, in the case of an intra prediction mode in which a mixedtype value is defined as 1, the mixed secondary transform set may beconfigured according to the method described in Embodiment 6 above. Inaddition, in the case of an intra prediction mode in which the mixedtype value is defined as 0, the secondary transform set may beconfigured according to the convention method (i.e., the methoddescribed in FIGS. 12 to 14 above).

Table 5 shows a case in which a method for configuring two types oftransform sets is used, but the present disclosure is not limitedthereto. That is, two or more mixed types indicating a method forconfiguring the transform set including the mixed secondary transformset may be configured (or defined). In this case, mixed type informationmay be configured as various N (N>2) types of values.

Furthermore, in an embodiment, the encoder/decoder may determine whetherthe transform set suitable for the current block is to be configured asa normal type or a mixed type by considering the intra prediction modeand the size of the current coding block (or transform block). Here, thenormal type indicates a secondary transform set configured according tothe convention method (i.e., the method described in FIGS. 12 to 14above). For example, when the mixed type (or mode type) value is 0, theencoder/decoder may configure the secondary transform set by applyingthe method described in FIGS. 12 to 14, and when the mixed type value is1, the encoder/decoder may configure the mixed secondary transform setincluding transform kernels of various sizes according to the size ofthe current block.

FIG. 22 is a diagram illustrating a method for determining a transformtype applied to secondary transform according to an embodiment to whichthe present disclosure is applied.

Referring to FIG. 22, the decoder is mainly described for convenience ofdescription, but the present disclosure is not limited thereto and themethod for performing the secondary transform through the determinationof the transform type may be substantially equally applied even to theencoder.

The decoder performs inverse quantization on a quantized transform block(or transform coefficient) (S2201).

The decoder determines (or selects) a secondary transform set (ortransform type set) used for secondary transform of a current blockbased on an intra prediction mode and/or a size of the current block(S2202). In this case, various secondary transform sets may bepredefined, and the method of configuring the secondary transform setdescribed in the embodiment and/or Embodiment 6 may be applied.

As an embodiment, the decoder may determine whether to use a mixed type(or a mixed secondary transform set) to configure the secondarytransform set based on the intra prediction mode of the current block.When the mixed type is used, the mixed secondary transform set accordingto the method described in Embodiment 6 above may be used.

The decoder performs the secondary transform for the dequantizedtransform block (or transform coefficient) by using the determinedtransform kernel (S2203). In this case, the decoder may parse asecondary transform index indicating the transform kernel used for thesecondary transform of the current block within the secondary transformset determined in step S2202. In this case, a step of parsing thesecondary transform index may be included in step S2203.

Embodiment 8: Method for Encoding Secondary Transform Index

In an embodiment, proposed is a method for efficiently encoding/decodingthe secondary transform index signaled from the encoder in the case ofconfiguring the secondary transform set based on the intra predictionmode and/or the size of the coding block (or transform block).

As described above, when the secondary transform is applied, theencoder/decoder may determine (or select) the secondary transform setapplied to the current coding block according to a preconfiguredcondition. In addition, the decoder may derive the transform kernelapplied to the current coding block within the determined secondarytransform set by using the secondary transform index signaled from theencoder. Here, the secondary transform index represents a syntax elementindicating the transform kernel applied to the secondary transform ofthe current block within the secondary transform set. In the presentdisclosure, the secondary transform index may be referred to as an NSSTindex, an LFNST index, etc.

As described above in Embodiment 6, the encoder/decoder may configurethe secondary transform set by using transform kernels of various sizes,and the number of transform kernels included in the secondary transformset is not fixed, but may be variably determined.

Therefore, since the number of available transform kernels may bedifferent for each secondary transform set, the encoder/decoder mayperform binarization on the secondary transform index by using atruncated unary method for efficient binarization. As an embodiment, theencoder/decoder may perform truncated unary binarization according tothe available maximum value of the secondary transform index for eachsecondary transform set by using Table 6 below.

TABLE 6 Binarization Binarization Binarization Binarization 1 2 3 4 NSST(maximum (maximum (maximum (maximum Index index: 2) index: 3) index: 4)index: 5) . . . 0  0  0   0    0 . . . 1 10  10  10   10 . . . 2 11 110 110  110 . . . 3 N.A 111 1110  1110 . . . 4 N.A 1111 11110 . . . 5 N.A11111 . . . . . . N.A . . .

In Table 6, the embodiment is described by assuming that the NSST isused as the secondary transform, but the present disclosure is notlimited to such a name. As described above, the secondary transformaccording to an embodiment of the present disclosure may be referred toas Non Separable Secondary Transform (NSST), Low-Frequency Non-SeparableTransform (LFNST), etc.

Referring to Table 6, the NSST index may be binarized by the truncatedunary binarization method. In this case, a binarization length may bedetermined according to a maximum index value within the secondarytransform set.

In an embodiment, the secondary transform index of Table 6 isencoded/decoded based on a context, and in this case, context modelingmay be applied in consideration of the following variables.

-   -   Size of coding block (or transform block)    -   Intra prediction mode    -   Mixed type value    -   Secondary transform index value of secondary transform set

Embodiment 9: Reduced Transform

In the embodiment of the present disclosure, various embodiments of thereduced transform proposed in FIGS. 15 to 22 are described in order toimprove a complexity problem of the transform. As described above, thereduced transform proposed in the present disclosure may be appliedregardless of a primary transform (e.g., DCT, DST) or a secondarytransform (e.g., NSST, Low-Frequency Non-Separable Transform (LFNST)).

FIG. 23 is a diagram illustrating a reduced transform structure based ona reduced factor as an embodiment to which the present disclosure isapplied.

Referring to FIG. 23, the decoder is mainly described for convenience ofdescription, but the reduced transform proposed in the embodiment may beequally applied to the encoder.

The decoder may apply the inverse reduced transform to the dequantizedtransform coefficients. In this case, the decoder may use apredetermined (or predefined) reduced factor (e.g., R or R/N) and/or atransform kernel in order to perform the reduced transform.

In one embodiment, the transform kernel may be selected based onavailable information such as the size (e.g., width/height) of thecurrent block (coding block or transform block), an intra/interprediction mode, CIdx, and the like. When a current coding block is aluma block, CIdx may have a value of 0. If not (i.e., if the currentcoding block is a Cb or Cr block), CIdx may have a nonzero value such asone.

FIG. 24 is a diagram illustrating a method for performing decoding byadaptively applying reduced transform as an embodiment to which thepresent disclosure may be applied.

Referring to FIG. 24, the decoder is mainly described for convenience ofdescription, but the method for performing the transform by using thereduced transform proposed in the embodiment may be equally applied tothe encoder.

The decoder performs the dequantization for the current block (S2401).

The decoder checks whether the transform is applied to (or used for) thecurrent block (S2402). If no transform is applied to the current block,the decoder terminates a transform process.

The decoder parses the transform index indicating a transform kernelapplied to the current block from the video signal when the transform isapplied to the current block (S2403).

The decoder checks whether a reduced inverse transform condition issatisfied (S2404). If the reduced inverse transform condition is notsatisfied, the decoder performs a normal inverse transform for thecurrent block (S2405). If the reduced inverse transform condition issatisfied, the decoder performs the reduced inverse transform for thecurrent block (S2407). In this case, the decoder may select thetransform kernel applied to the current block based on the transformindex parsed in step S2403 (S2406). As the embodiment, the transformkernel may be selected based on available information such as the size(e.g., width/height) of the current block (coding block or transformblock), the intra/inter prediction mode, CIdx, and the like. Further,when the reduced inverse transform is applied to the current block, stepS2406 may include selecting the reduced factor.

In an embodiment, the reduced inverse transform condition may be appliedto the above-described conditions of 6) (e.g., Tables 3 and 4). In otherwords, whether to apply the reduced inverse transform may be determinedbased on the size of the current block (coding block or transform block)and the transform type (or transform kernel).

As an example, the reduced transform may be used when the followingspecific condition is satisfied. In other words, the reduced transformmay be applied to blocks having a predetermined size or larger (orlarger than the predetermined size) that satisfy the following specificcondition.

-   -   Width>TH && Height>TH (where TH is a predefined value (e.g. 4),        which represents a specific threshold)

or

-   -   Width*Height>K && MIN (width, height)>TH (where K or TH is a        predefined value, and represents a specific threshold)

As another example, the reduced transform may be used when the followingspecific condition is satisfied. In other words, the reduced transformmay be applied to blocks having a predetermined size or smaller (orsmaller than the predetermined size) that satisfy the following specificcondition.

-   -   Width<=TH && Height<=TH (where TH is a predefined value (e.g.        8), which represents a specific threshold)

or

-   -   Width*Height<=K && MIN (width, height)<=TH (where K or TH is a        predefined value, and represents a specific threshold)

As another example, the reduced transform may be applied only to apredetermined block group.

-   -   Width==TH && Height==TH

or

-   -   Width==Height

As the embodiment, if a usage condition of the reduced transform is notsatisfied, a normal transform may be applied. Specifically, the regulartransform may be predefined and available to the encoder/decoder. Thefollowing shows an example of the normal transform.

-   -   DCT2, DCT4, DCT5, DCT7, DCT8

or

-   -   DST1, DST4, DST7,

or

-   -   Non separable transform

or

-   -   NSST(HyGT)

or

-   -   LFNST (Low-Frequency Non-Separable Transform)

The above conditions may be interpreted based on logical operators asshown in Table 7 below.

TABLE 7 Logical operators The following logical operators are defined asfollows: x && y Boolean logical ″and″ of x and y. x | | y Booleanlogical ″or″ of x and y. ! Boolean logical ″not″. x ? y : z If x is TRUEor not equal to 0, evaluates to the value of y; otherwise, evaluates tothe value of z.

In addition, as illustrated in FIG. 24, the reduced transform conditionmay depend on a transform index Transform_idx indicating the transformapplied to the current block. As an example, the Transform_idx may betransmitted from the encoder to the decoder twice. One may be ahorizontal transform index Transform_idx_h and the other one may be avertical transform index Transform_idx_v.

FIG. 25 is a diagram illustrating a method for performing decoding byadaptively applying reduced transform as an embodiment to which thepresent disclosure may be applied.

Referring to FIG. 25, the decoder is mainly described for convenience ofdescription, but the method for performing the transform by using thereduced transform proposed in the embodiment may be equally applied tothe encoder.

In an embodiment of the present disclosure, the above-described reducedtransform may be used for the secondary transform. In this regard, adescription duplicated with the method described in FIG. 24 will beomitted.

1) The decoder performs the dequantization for the current block andthen checks whether the NSST is activated in the current block. Thedecoder may determine whether it is necessary to parse the NSST indexusing a predefined condition.

2) If the NSST is activated, the decoder parses the NSST index andchecks whether the reduced secondary inverse transform is applied.

3) The decoder checks whether the reduced inverse transform condition issatisfied.

4) If the reduced inverse transform condition is not satisfied, thedecoder performs a normal inverse transform for the current block.

5) If the reduced secondary inverse transform condition is satisfied,the decoder performs the reduced secondary inverse transform for thecurrent block.

6) In this case, the decoder may select the transform kernel applied tothe current block based on the NSST index. As the embodiment, thetransform kernel may be selected based on available information such asthe size (e.g., width/height) of the current block (coding block ortransform block), the intra/inter prediction mode, CIdx, and the like.Further, when the reduced secondary inverse transform is applied to thecurrent block, the decoder may select the reduced factor.

In an embodiment, whether to apply the reduced inverse transform may bedetermined based on the size of the current block (coding block ortransform block) and the transform type (or transform kernel).

Embodiment 10: Reduced Transform as a Secondary Transform with DifferentBlock Size

In an embodiment of the present disclosure, a reduced transformconsidering various block sizes used for the secondarytransform/secondary inverse transform is proposed. As an example, thereduced transforms for different block sizes of 4×4, 8×8, and 16×16 usedfor the secondary transform/secondary inverse transform may be defined.

FIGS. 26 and 27 are diagrams illustrating examples of forwarded reducedsecondary transform and inverse reduced secondary transform and a pseudocode for deriving the same.

Referring to FIGS. 27 and 28, the reduced secondary transform and thereduced secondary inverse transform when the block to which thesecondary transform is applied is an 8×8 block and a reductioncoefficient R is 16. The reduced secondary transform and the reducedsecondary inverse transform illustrated in FIG. 26 may be derived byusing the pseudo code illustrated in FIG. 27.

Embodiment 11: Reduced Transform as a Secondary Transform withNon-Rectangular Shape

As described above, due to the complexity problem of the secondarytransform to which the non-separable transform is applied, in an imagecompression technique in the related art, the secondary transform isapplied to the top-left 4×4 or 8×8 region of the coding block (ortransform block).

The embodiment of the present disclosure proposes a method for applyingthe reduced secondary transform to various non-square figures inaddition to a 4×4 or 8×8 square region.

FIG. 28 is a diagram illustrating a method for applying reducedsecondary transform to a non-square region as an embodiment to which thepresent disclosure is applied.

Referring to FIG. 28, in an embodiment, the reduced secondary transformmay be applied to only a portion of the block, as illustrated in FIG.29.

In FIG. 28, each square represents a 4×4 region. Accordingly, theencoder/decoder may apply the reduced secondary transform to a 10×4pixel, i.e., 160 pixel region. In this case, the reduction coefficientR=16 and the entire RST matrix corresponds to a 16×160 matrix, therebyreducing calculation complexity of applying the secondary transform.

Embodiment 12: Reduced Factor

FIG. 29 is a diagram illustrating reduced transform controlled by areduction factor as an embodiment to which the present disclosure isapplied.

Referring to FIG. 29, as described above, the reduction transformaccording to the embodiment of the present disclosure may be controlledby the reduction factor, as illustrated in FIG. 29.

Specifically, modifying the reduction factor may modify memorycomplexity and the number of multiplication operations. As mentionedpreviously in FIG. 15 and Equation 6 as the reduction factor R/N, thememory and multiplication may be reduced by modifying the reductionfactor. For example, for the 8×8 NSST with R=16, the memory and themultiplication may be reduced by ¼.

Embodiment 13: High Level Syntax

The embodiment of the present disclosure proposes a high-level syntaxstructure for controlling the reduced transform at a high level.

In an embodiment, as shown in the example of Table 8 below, whether thereduced transform is accepted through a sequence parameter set (SPS) andinformation on a size and/or reduction factor may be transmitted.However, the present disclosure is not limited thereto and the syntaxmay be signaled through a picture parameter set (PPS), a slice header,etc.

TABLE 8 seq_parameter_set_rbsp( ) { Descriptor   sps_video_parameter_set_id u(4)    sps_max_sub_layers_minus1 u(3)   sps_temporal_id_nesting_flag u(1)    profile_tier_level(sps_max_sub_layers_minus1 )    sps_seq_parameter_set_id ue(v)   chroma_format_idc ue(v)    if( chroma_format_idc == 3)      separate_colour_plane_flag u(1)    pic_width_in_luma_samples ue(v)   pic_height_in_luma_samples ue(v)    conformance_window_flag u(1)   if( conformance_window_flag ) {       conf_win_left_offset ue(v)      conf_win_right_offset ue(v)       conf_win_top_offset ue(v)      conf_win_bottom_offset ue(v)    } ...Reduced_transform_enabled_flag u(1) If(reduced_transform_enabled_flag) {  reduced_transform_factor ue(v)   min_reduced_transform_size ue(v)  max_reduced_transform_size ue(v)   reduced_transform_size ue(v) }   sps_extension_flag u(1)    if( sps_extension_flag )       while(more_rbsp_data( ) )          sps_extension_data_flag u(1)   rbsp_trailing_bits( ) }

Referring to Table 8, if Reduced_transform_enabled_flag is 1, thereduced transform may be available and applied. In a case whereReduced_transform_enabled_flag is 0, the case may indicate that thereduced transform may not be available. IfReduced_transform_enabled_flag does not exist, the value may beestimated to be equal to zero.

Reduced_transform_factor represents a syntax element that specifies thenumber of reduced dimensions for the reduced transform.

min_reduced_transform_size represents a syntax element that specifies aminimum transform size to which the reduced transform is to be applied.If min_reduced_transform_size does not exist, the value may be estimatedto be equal to zero.

max_reduced_transform_size represents a syntax element that specifies amaximum transform size to which the reduced transform is to be applied.If max_reduced_transform_size does not exist, the value may be estimatedto be equal to zero.

Reduced_transform_factor represents a syntax element that specifies thenumber of reduced dimensions for the reduced transform. IfReduced_transform_factor does not exist, the value may be estimated tobe equal to zero.

Embodiment 14: Secondary Transform Kernels

The embodiment of the present disclosure proposes various secondarytransform kernels.

In an embodiment, a 4×4 NSST kernel for a DC mode may be defined asshown in Table 9 below.

TABLE 9 { 223, −84, −36, 3, −79, 26, 16, −2, −18, 8, 5, −1, −2, 0, −1, 1}, { 76, 216, −82, −32, 8, −51, 10, 8, −27, −39, 17, 10, 2, 3, −1, −1 },{ −54, 39, 20, −8, −215, −6, 71, 2, 88, −14, −31, 2, 15, 2, −7, −1 }, {6, −47, −4, 13, 7, −229, 39, 50, 1, 69, −9, −20, −3, 35, −2, −12 }, {58, 40, 203, −108, 4, −19, −51, 26, −6, −1, −56, 21, −7, −2, 13, −3 }, {36, 8, −33, 0, 65, 9, −9, −2, 196, 10, −64, −3, −120, 5, 42, 0 }, { 18,27, 47, 22, 65, 37, 206, −50, −27, 60, −62, −9, −3, −34, −37, 16 }, { 1,51, −10, 26, −32, 30, −73, 16, −4, 194, −8, −72, 19, −98, 32, 28 }, {30, 43, 83, 209, −10, −25, −46, −52, −6, −53, −30, −47, −5, 26, 9, 6 },{ 29, 3, −6, 2, 50, 7, −21, −1, 107, 11, −29, 1, 205, 35, −77, −9 }, {17, 19, 68, 19, 13, 21, 37, 74, 63, 10, 197, −51, −35, 1, −76, −5 }, {4, −2, 16, −29, −9, −55, −28, −203, 31, 37, 84, 75, −9, −20, −23, 58 },{ 7, −26, 2, 11, 16, −51, 11, 35, 27, −100, 1, 8, 33, −213, 10, 58 }, {−6, −14, −15, −73, −1, −11, −10, −47, −9, −51, −28, −196, −13, 33, −47,109 }, { −6, 1, −20, 18, −16, −3, −55, 24, −21, 14, −69, 50, −76, −29,−215, −2 }, { 0, 5, 2, 30, 1, 15, 7, 78, −2, 19, −1, 88, 5, 64, 19, 213},

Further, in an embodiment, a 4×4 NSST kernel for a planar mode may bedefined as shown in Table 10 below.

TABLE 10 { −211, 122, 2, −2, 66, −27, −10, 3, 21, −20, 2, 1, 4, −1, 1, 0}, { 116, 169, −116, −7, 32, −82, 13, 16, −28, −15, 31, −5, −3, 5, −2,−2 }, { −17, 77, −31, −7, −203, 71, 37, −4, 95, −32, −17, 3, 11, −16, 0,3 }, { −32, −60, 4, −2, −57, −201, 101, 25, 53, 49, −4, −16, −10, 23,−18, −8 }, { 60, 81, 186, −108, −2, −38, −46, 18, 35, 1, −50, 23, −26,8, 5, −2 }, { 31, −37, −44, 8, 92, 9, 1, −5, 187, −95, −33, 9, −82, 16,24, 0 }, { 15, 43, −3, 15, 58, 63, 110, −49, 27, 157, −110, −24, −24,−44, −9, 28 }, { −3, 12, −18, 94, −40, −28, −164, 48, 39, 115, −12, −49,−70, 2, 51, −11 }, { −27, −46, −104, −190, 6, 17, −44, 83, 15, 64, −19,25, 20, −27, 11, −5 }, { 25, −3, 17, 37, 30, −34, −45, −1, 83, 3, 2,−17, 189, −125, −15, 25 }, { 11, 15, 42, −22, 28, 58, 39, 38, 69, 67,189, −77, −7, 30, −57, −26 }, { 9, 9, −11, 31, 15, 27, −14, 58, 22, 13,−85, −3, 103, 178, −80, −56 }, { 3, 4, 29, 42, 13, 29, 71, 197, −34,−53, −41, −65, −11, −61, 55, −32 }, { 1, 3, 3, 54, 0, 5, −9, 74, 5, 19,13, 164, −54, −62, −151, 10 }, { 6, 9, 12, 27, 10, 9, 37, −7, 21, 48,43, 138, 40, 10, 137, −138 }, { −2, −4, −11, −20, −4, −10, −22, −57,−13, −25, −37, −69, −32, −77, −86, −200 },

Further, in an embodiment, an 8×4 NSST kernel for the DC mode may bedefined as shown in Table 11 below.

TABLE 11 { 218, −83, −33, 0, −5, −1, −2, 0, −89, 37, 13, 0, 2, 0, 1, 0,−18, 8, 3, −1, 1, 0, 0, 0, −4, 1, 0, 1, 0, 0, 0, 0, −4, 2, 1, 0, 0, 0,0, 0, −2, 0, 0, 0, 0, 0, 0, 0, −2, 1, 0, 0, 0, 0, 0, 0, −1, 0, 0, 0, 0,0, 0, 0 }, { 77, 188, −115, −21, −12, −3, −5, −1, 40, −65, 23, 6, 0, 1,0, 0, −47, −27, 27, 7, 2, 1, 1, 1, 4, 1, −3, −1, 1, 0, 0, 0, −8, −5, 5,1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, −3, −2, 2, 0, 0, 0, 0, 0, 0, 0,0, 0, 0, 0, 0, 0 }, { 45, −76, −3, 20, −2, 4, −1, 1, 197, −6, −68, −4,−6, −2, −3, 0, −102, 46, 29, −8, 3, −1, 1, 0, −8, −7, 8, 2, 0, 0, 0, 0,−14, 5, 1, 0, 0, 0, 0, 0, −3, −1, 2, 0, 0, 0, 0, 0, −5, 2, 1, 0, 0, 0,0, 0, −1, 0, 1, 0, 0, 0, 0, 0 }, { 70, 75, 181, −106, −6, −17, 2, −6,20, −1, −72, 31, 3, 3, −1, 1, 7, −34, −39, 22, 3, 3, 0, 1, −16, 8, 11,−5, 0, 0, 0, 0, −2, −5, −6, 5, 1, 1, 0, 0, −3, 1, 2, −1, 0, 0, 0, 0, −1,−2, −2, 2, 0, 0, 0, 0, −1, 0, 0, 0, 0, 0, 0, 0 }, { −17, 39, −34, −8, 4,−2, 1, 0, 11, 212, −53, −45, −5, −8, −1, −3, −17, −91, 52, 22, −1, 3, 1,1, 3, −22, 0, 7, 1, 1, 0, 0, 2, −6, 1, −1, 0, 0, 0, 0, 0, −6, 1, 2, 0,1, 0, 0, 0, −2, 1, 0, 0, 0, 0, 0, 0, −2, 0, 1, 0, 0, 0, 0 }, { 38, −6,−58, 8, 2, 2, 0, 1, 83, 8, −25, −4, −5, 0, −2, 0, 191, −14, −46, −2, −7,−1, −3, 0, −108, 20, 38, −3, 3, 0, 2, 0, 6, −7, 2, 2, 0, 0, 0, 0, −19,4, 5, 0, 1, 0, 0, 0, 3, −2, 1, 0, 0, 0, 0, 0, −6, 1, 1, 0, 0, 0, 0, 0 },{ 21, 6, 54, −2, −11, 0, −1, 0, 75, 47, 203, −71, −25, −7, −2, −2, −12,−11, −70, 32, 11, 2, 1, 1, −7, −7, −28, 8, 6, 1, 0, 1, −7, 1, 0, 0, 0,1, 0, 0, −1, −2, −7, 3, 2, 0, 0, 0, −2, 0, −1, 0, 0, 0, 0, 0, −1, −1,−2, 1, 1, 0, 0, 0 }, { 25, 82, 58, 165, −84, 19, −15, 4, −17, 36, −36,−38, 30, −7, 4, −1, 5, 78, −47, −62, 17, −5, 3, −2, 1, −39, 24, 13, −7,3, 0, 1, −3, −5, −4, −1, 4, 0, 1, 0, 1, −5, 4, 1, 0, 0, 0, 0, −1, −1,−2, −2, 1, −1, 0, 0, 0, −2, 1, 0, 0, 0, 0, 0 }, { 13, −31, 35, 112, −40,13, −8, 3, 9, −66, −8, −13, 17, 0, 2, 0, −11, −187, 20, 23, 5, 8, 1, 2,2, 80, −24, −18, −1, −3, 0, −1, 1, 6, 2, −9, 1, −1, 0, 0, 0, 10, −2, 0,−1, 0, 0, 0, 0, 0, 0, −2, 1, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0 }, { 17,−5, −23, −4, 1, 1, −1, 0, 34, 10, −33, 11, 1, −1, 0, 0, 74, −4, −67, 5,3, 0, −1, 1, 176, 3, −65, −7, 1, −2, −2, 0, −110, 23, 51, −8, 2, 0, 2,0, 34, −6, −8, 2, −1, 0, −1, 1, −19, 3, 7, −1, 0, 0, 0, 0, 6, −2, −1, 0,0, 0, 0, 0 }, { 12, 14, 33, 39, 14, −4, 0, −1, 26, 46, 60, 174, −64, 15,−9, 3, 28, −9, 78, −106, 16, −1, 3, −1, −5, 3, −51, −8, 10, 1, 2, 0, −6,−5, 5, 8, −1, 1, 1, 0, −1, 1, −8, −6, 3, −1, 0, 0, 0, −1, 1, 1, 0, 1, 0,0, 0, 0, −1, −2, 1, 0, 0, 0 }, { 22, 6, 53, −12, −13, 1, 0, 0, 9, −37,14, −99, 23, −5, 4, −1, 79, 32, 179, −3, −47, −1, −6, −1, 14, −26, −62,53, 8, 1, 0, 1, −24, 5, −10, −7, 7, 0, 0, 0, 5, −3, −5, 7, −2, 1, 0, 0,−3, 1, −2, −1, 1, 0, 0, 0, 1, −1, −1, 3, −1, 0, 0, 0 }, { 21, 31, 27,80, 205, −58, 11, −6, −8, −14, −20, −51, −69, 23, −2, 1, −6, −2, −18,−9, −41, 12, −3, 0, 5, 2, 9, 10, 16, −5, 1, 0, −1, −1, 0, −7, −8, 3, 1,0, 1, 1, 1, 4, 3, −1, 0, 0, 0, −1, 0, −2, −4, 1, 0, 0, 0, 1, 0, 1, 2, 0,0, 0 }, { −5, 18, −4, −8, 0, −3, 1, −1, −8, 34, −19, −30, 16, −5, 1, 0,−16, 71, −8, −9, −2, −3, 1, −1, −27, 176, −103, −23, 19, −9, 3, −3, 22,−93, 57, 10, −7, 5, −2, 2, −7, 28, −14, 0, 1, −2, 1, 0, 3, −12, 6, 0, 1,0, 0, 0, −2, 4, −2, 1, −1, 0, 0, 0 }, { 6, 5, 7, 15, 80, −8, 2, −3, 21,22, 38, 54, 215, −39, 5, −3, −1, −2, −1, −5, −48, 25, −2, 1, −2, −4, 1,−10, −49, 11, 3, 0, −1, 2, −1, 8, −3, −3, 1, 1, 0, −1, 0, −3, −9, 1, 0,0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, −3, 1, 0, 0 }, { −4, −16, −14,−64, 7, −6, 4, −1, 1, −17, −6, −77, 14, −8, 5, 0, −16, −60, −40, −201,22, −6, 4, 0, 0, 6, −9, 54, −41, 10, −3, −1, 6, 10, −2, 53, −2, −4, 1,−1, −1, 0, −2, 0, 3, −1, −1, 0, 2, 0, 1, 5, −1, 0, −1, 0, −1, 0, 0, 0,−1, −1, 0, 0 }, { 1, 1, 11, −5, −4, 2, 0, 1, 7, 14, 37, −8, −16, 2, 0,1, 13, 14, 48, −14, −12, 0, 0, 0, 68, 96, 168, −24, −39, −1, −4, −1,−63, −71, −65, 40, 14, 0, 3, 0, 19, 16, 10, −10, −1, 0, 0, 0, −8, −8,−5, 5, 0, 1, 0, 0, 3, 3, 1, −1, 0, 0, 0, 0 }, { 21, 6, −6, 0, −1, 1, −1,0, 32, 9, −3, −2, −2, −1, 0, 0, 37, 14, −3, 0, 2, −2, −1, 0, 116, 33,−13, −13, −4, 1, −1, 1, 191, 42, −88, −6, 2, −2, −2, 1, −33, 2, 18, −1,0, 0, 1, 0, 12, 2, −4, 2, 0, 0, 0, 0, −6, 2, 3, 0, 0, 0, 0, 0 }, { 7,11, 11, −3, 57, 206, −13, −10, −2, 0, −2, 3, −11, −33, 4, 5, −4, −6, −6,2, −29, −128, −6, 9, −1, 1, −1, 4, 8, 4, −11, −3, −1, 1, 0, 14, −2, 1,1, −2, 0, 0, 1, −1, 3, 1, 0, 0, 0, 1, −1, 0, 0, −2, 1, 0, 0, 0, 1, 0, 0,0, 0, 1 }, { 3, 3, 4, 8, 45, 1, −3, 0, 1, −2, 0, 4, 24, 27, −4, −1, 15,11, 26, 41, 196, −38, 3, −1, 1, 21, 6, 92, −81, −23, −4, 1, −2, −2, 5,−38, −43, 2, −2, 0, 0, 4, −6, 3, 6, −1, 2, 0, 1, −2, 2, −3, −4, 1, 1, 1,1, 1, 0, 1, 1, 0, 1, 1 }, { 9, −18, 3, 7, −4, 2, 0, 1, 13, −25, 1, 8,−3, 3, −2, 0, 14, −33, −2, 13, −9, 4, 0, 0, 46, −100, 13, 24, −11, 2,−3, 1, 69, −187, 81, 38, −25, 10, −5, 3, −9, 28, −17, −6, 5, −1, 0, −1,3, −9, 2, −1, 1, 1, 0, 0, −3, 5, −5, 1, −1, 0, 0, 0 }, { 4, 3, 7, 10,19, −7, 0, 0, −3, −9, −7, −36, −9, −2, 0, −2, 14, 11, 29, 19, 72, −13,5, 1, −13, −39, −41, −165, −65, 39, 2, 2, −4, 21, 2, 139, −3, −14, 3,−4, 1, 0, 4, −17, 11, 2, −1, 1, 0, 0, −2, 4, −3, 1, −1, 0, −1, −1, −1,0, 1, −1, 0, 0 }, { −2, −3, −4, 3, −22, −63, 6, 11, 6, 9, 8, 1, 45, 175,−9, −15, −5, −7, −6, 1, −44, −148, 6, 15, 0, −3, −1, −4, 10, −36, −16,2, 0, 7, −1, 27, 0, 24, −3, −10, 0, −1, 1, −4, −2, −10, 1, 2, 1, 1, 0,0, 3, 6, 0, 0, 0, −1, 0, 1, −2, −4, 1, 1 }, { 3, 7, 18, −6, −12, 3, 1,1, 2, 6, 19, −10, −15, 2, 0, 1, 3, 11, 34, −13, −32, 3, 0, 1, 15, 31,84, −21, −45, −6, −4, −1, 53, 95, 189, −17, −54, −5, −2, 0, −3, −3, −4,8, 6, 1, 1 :1, 3, 5, 8, −2, 1, 2, 1, 1, 0, −4, −1, 4, −1, 1, 1, 0 }, {4, 0, −1, 0, 0, 0, −1, 0, 5, −1, −5, 1, 3, −1, 0, 0, 5, 0, −1, 0, 2, 0,−1, 0, −14, 7, 11, −8, 6, −7, −2, 0, 45, −5, −18, 8, 0, −1, −2, 1, 166,−71, −152, 54, 24, −5, −2, 1, −39, 28, 21, −16, −3, 1, 1, −1, −5, 3, 9,−4, −3, 1, 0, 0 }, { 0, 0, 0, 2, −2, 13, −2, −1, 1, 1, 1, 2, 22, −9, 2,0, 2, 0, 2, 1, 13, 37, −21, −3, 5, 0, 9, 12, 102, −133, −40, 5, −8, 16,−19, 87, −140, −44, 25, −2, 5, −9, 15, −31, 11, 25, −5, −2, −1, 5, −4,4, 1, −5, 2, 4, −1, 0, 1, 0, −5, 3, 4, −1 }, { −2, −2, 2, −9, 14, −6,−151, 109, 0, 0, 0, 2, −3, −7, 41, −23, 2, −1, 1, 3, −8, −3, 136, −85,−1, 1, −1, 2, 5, 3, −37, 23, −1, 1, −1, 2, −8, 6, −15, 13, 0, −1, 1, −3,6, −1, 4, −4, 0, −1, 0, 0, 0, −1, 1, 0, 0, 0, 0, 0, 0, 0, −1, 0 }, { −3,4, −7, 18, −24, 12, −6, 2, −4, 7, −12, 33, −47, 24, −8, 3, −8, 10, −24,55, −89, 38, −5, 1, −14, 23, −41, 108, −134, 27, 0, −6, −6, 24, −28, 99,−44, −22, 13, −3, 1, 4, −7, −2, 16, −8, 0, 0, 0, 2, −1, 0, 3, 0, 1, 0,−1, 1, −1, 1, −1, −1, 2, 0 }, { −1, −2, −2, 15, −17, −105, 54, 13, −1,−4, −2, 19, −21, −157, 65, 13, −1, −3, −1, 16, −15, −129, 10, 8, 0, 5,−1, 26, −8, −8, −38, −2, 0, 6, −2, 21, −14, 14, −12, −8, 1, 0, 0, −5, 5,10, 0, −2, 1, 0, 0, −1, 1, 3, 0, 0, 1, −1, 0, −1, 0, 2, 0, 0 }, { 2, 1,−1, −2, −1, 0, 0, −1, 3, 5, 0, −2, −2, 1, 0, 1, 2, 1, −2, −2, −2, 0, 1,0, −15, −15, −7, 2, −1, 3, 0, 1, 22, 24, −4, −16, −5, 0, 1, 0, 148, 180,60, −53, −29, −3, −1, −1, −25, −29, 5, 15, 4, 0, 0, 1, −14, −14, −4, 6,5, 0, 0, 0 }, { 1, 1, −1, 4, −7, 1, 72, −72, −1, −1, 0, −5, 9, −3, −121,118, 2, −1, 0, 2, −6, −10, 119, −95, 1, 0, 0, 1, 7, 7, −21, 4, −3, 3,−4, 4, −16, 3, −30, 33, 1, −3, 1, −4, 8, 2, 14, −15, 0, −1, 0, −2, 2,−3, −5, 5, 0, 0, −1, 2, −1, 1, 3, −3 }, { 1, 0, −1, −12, −12, 3, 2, 2,0, −1, −2, −18, −23, −1, 3, 3, −1, −3, −6, −32, −44, −11, −6, −3, −10,−23, −30, −88, −113, −62, −21, −5, −17, −39, −50, −132, −105, −24, −4,−4, −6, −11, −17, −28, −12, −3, 1, −1, −1, −1, 0, −6, −3, 2, 2, 0, 0, 3,1, 0, −3, 1, 0, −1 }, { −1, 2, −1, 0, −1, 1, 0, −1, −2, 3, −4, −1, 6,−3, 2, 1, −2, 2, 0, −1, −1, 2, −2, −1, 10, −14, 13, −8, 14, −13, −2, 0,−22, 34, −20, 3, −3, −2, 0, −1, −88, 138, −155, 37, 48, −19, 8, −5, 39,−55, 49, −9, −14, 7, −3, 1, 9, −10, 9, 0, −9, 3, 0, 0 }, { 1, 1, 1, 3,6, 12, 0, −1, 0, 0, −1, 2, −12, −12, −4, 4, −1, 2, −1, 5, −6, 19, 5, −4,−5, −9, −10, 1, −62, −180, −9, 5, 11, 6, 22, 23, 144, −5, −52, 3, −2, 2,−7, −14, −18, 45, 13, −11, 1, −1, 2, 11, −1, −14, 5, 4, 1, −1, 0, −5, 4,4, −1, 2 }, { −1, 0, 2, −1, 0, 1, 0, 0, −2, 0, 0, 0, 1, 0, 0, −1, 0, 0,1, −1, −1, 0, 0, −1, −6, 1, 2, 0, 0, −1, 0, −1, 11, −2, −9, 3, −1, 4, 2,0, −25, 3, 19, −3, 2, 5, 1, −1, −118, 30, 126, −28, −29, 0, 1, 1, 121,−34, −120, 32, 27, −4, −1, 0 }, { −3, −3, −2, −1, 0, −8, −73, −122, 0,0, 0, −1, −1, 2, 32, 24, 3, 4, 2, 1, 0, 15, 109, 162, 0, 0, 0, 2, 4, 3,−36, −32, 0, −1, −2, −2, −6, −4, −24, −47, 0, −1, −1, −3, 6, 8, 7, 6, 0,1, 0, −1, 3, 3, 2, 4, 1, −1, 1, 0, −1, 0, −1, 0 }, { 3, 4, 2, 1, 3, 46,129, 82, 3, 4, 2, −1, 5, 59, 148, 79, 1, 2, 1, 0, 8, 48, 58, 19, −1, −1,−1, −3, 5, 10, −30, −17, −1, −1, −1, −3, 3, −4, −22, −15, 0, −1, 1, 0,3, −1, −2, −2, 0, 0, 0, 0, 1, 0, −1, 0, 0, 0, 0, 1, 0, 0, −1, 0 }, { −1,−1, −1, −2, −1, 0, 0, −1, 1, 0, 3, 5, 1, −3, 0, 1, −1, −2, −1, −5, −3,−1, 0, −1, −2, −7, −8, −20, −14, −35, −11, −2, −4, −11, −7, −1, −10,−13, −4, 0, 16, 33, 92, 213, 73, −16, −6, 2, −4, −4, −19, −18, 8, 14, 3,−1, 1, −7, −15, −26, −11, 4, 2, 0 }, { −1, −2, −1, 0, 0, 0, 0, 0, 0, 0,0, 0, 0, 1, 1, 0, 1, 0, −1, 0, 0, −1, 0, 0, −4, −4, −1, 2, 2, 1, 1, 1,10, 11, 4, 0, −2, −3, −3, −1, −8, −11, −7, −1, 0, −2, 0, −1, −104, −132,−60, 29, 31, 6, 0, 0, 105, 128, 47, −34, −26, −3, 0, −1 }, { 0, 0, 0,−1, −2, 10, −9, 0, 0, 0, −1, −1, 2, −12, 14, 3, 0, −1, 0, −2, −4, 16,−10, 6, 0, −1, −1, −3, 3, −34, 93, −21, −6, 11, −12, 17, −43, 156, −113,18, 8, −10, 17, −33, 70, −80, 26, 2, −1, 1, −4, 2, −12, 17, 7, −8, −2,1, 1, −2, 2, 5, −14, 4 }, { −3, −4, −2, −2, −1, −16, −90, −100, 3, 4, 3,2, 1, 24, 116, 127, −2, −2, 0, 1, 0, −10, −83, −83, 0, −1, −1, −2, −4,0, 4, −9, 2, 0, 4, 1, 11, 1, 27, 43, −1, −1, 0, 1, −5, −3, −14, −15, 0,0, 1, 1, −1, 0, 5, 3, 0, 1, 0, 0, 1, 0, −2, −2 }, { 0, 0, −1, 2, −2, 5,−1, −2, −1, 0, −1, 2, −7, 2, 3, 1, 0, 1, 0, 1, 1, 8, −1, 3, 0, −1, −1,−5, −1, 7, 1, 11, −4, 6, −9, 22, −48, 86, −20, −4, −12, 12, −25, 87,−173, 47, 45, −22, −3, 3, −10, 65, −43, −58, 24, 5, 2, −1, −5, −7, 26,−20, −1, 9 }, { 0, 1, −1, 0, 1, −1, 0, 0, −1, 0, 0, 0, −1, 0, 0, 1, 0,0, −1, 0, 1, 0, 0, −1, −3, 3, −2, 0, −1, 1, 0, 0, 5, −6, 3, −2, 1, −2,−1, 1, −20, 27, −15, −4, 6, −7, 2, 0, −58, 83, −84, 19, 14, −7, 6, −4,101, −143, 116, −7, −38, 17, −6, 6 }, { 0, 1, 0, 0, 1, 7, −5, −9, 0, 1,0, 0, 2, 12, −16, −16, −1, 2, −1, −1, 3, 27, −73, 6, −4, 4, −4, −1, 0,44, −195, 71, −2, 3, 0, −5, 8, 43, −89, 27, 3, −2, 5, −14, 37, 5, 24,−13, 1, −1, 2, −3, 8, −10, 9, 2, −1, 0, −1, −1, −3, 2, 1, 1 }, { −3, −1,3, 0, −1, 0, 1, −1, −4, −1, 4, 0, −1, 0, 0, 0, −2, −1, 2, 1, 0, 0, 1, 0,0, −2, −1, 1, −2, 0, −1, 1, −1, −5, −1, 3, 0, 2, 2, 0, −27, −6, 34, 8,−8, 1, 6, −2, −99, 4, 137, −6, −36, 2, 4, 2, −101, 15, 144, −12, −45, 7,4, 1 }, { 1, 0, −1, 4, −4, 15, 60, −126, 1, −1, 0, 4, −4, 15, 85, −160,1, 0, 0, 2, −2, 9, 57, −84, 0, 1, −1, 2, −3, −2, 4, 32, −1, 0, −1, 1,−4, −4, −3, 30, 0, −1, 0, 0, 0, −7, −2, 2, 2, 0, −1, 1, −2, −1, −1, 2,0, −1, 0, 0, 0, 0, −2, 1 }, { 0, 0, 0, −1, 4, 2, −3, 0, 0, 0, 0, 1, −5,−1, 5, −1, 0, 0, −1, −1, 5, 3, −5, 0, 1, 0, 1, −2, 0, −4, 15, 5, −2, 3,−6, 14, −10, 21, −26, 0, −4, 4, −2, 17, −72, 5, 18, −14, 7, −8, 12,−132, 156, −6, −30, 13, 0, −1, 0, 83, −86, 15, 10, −9 }, { 0, 0, 0, 1,1, −3, −5, 1, 0, 0, 1, 2, 0, −6, −2, 2, 0, 0, 0, 2, 6, −5, −12, −5, 1,1, 3, 6, 14, −6, −18, −21, 0, −1, 5, 16, 6, −103, −79, 14, −5, −4, −4,13, −98, −148, 36, 52, −2, 1, −8, −23, −32, 83, 29, −14, 2, 1, 3, −1,33, 21, −7, −3 }, { 0, 1, 1, 0, 2, 10, 12, 1, −1, −1, −1, −1, −4, −13,−14, −5, 1, 2, 2, 1, 4, 21, 23, 12, −3, −4, −4, −6, −15, −41, −85, −49,5, 6, 4, 6, 26, 109, 150, 43, −3, −4, −1, −1, −32, −104, −36, 23, −1, 0,−2, −7, 7, 24, −19, −19, 1, 3, 1, 3, 1, 4, 9, 1 }, { −3, 4, −1, −1, 1,−2, 1, −1, −4, 4, −1, −2, 2, 0, 1, 0, −2, 2, 1, −2, 2, −2, 1, −1, 1, −1,3, −2, 3, −2, 1, 0, −5, 6, 0, −3, 2, −3, −2, 1, −46, 60, −34, −4, 13,−6, 0, 0, −108, 140, −80, −14, 37, −13, 6, −6, −78, 96, −46, −15, 31,−12, 4, −4 }, { 0, 1, 1, 2, 3, 0, −1, 0, 0, −1, −1, −3, −3, −2, 0, 1, 1,1, 1, 3, 3, 0, −1, −1, 0, 0, 1, 1, 1, 3, 4, 1, 0, −1, −1, −2, −3, −11,−12, −1, −3, −6, −8, −7, −18, −27, −8, −2, 11, 24, 61, 123, 116, 32, 2,0, −9, −22, −59, −134, −89, 0, 7, 4 }, { 0, 0, 0, 0, 0, −5, 0, 19, 0, 0,0, 0, 1, 6, −2, −28, 0, 0, 0, 0, 0, −9, 2, 39, 0, −1, 1, 0, 5, 14, −5,−135, −2, 1, −2, −1, −6, −37, −28, 191, 0, 1, 0, 1, 6, 38, 11, −51, 1,−2, 2, 3, 1, −34, 0, −6, 0, 0, −2, −4, −5, 7, −2, 12 }, { 3, 3, 1, −1,−1, −1, 0, 0, 3, 3, 1, −1, −1, 0, −1, 1, 1, 1, 0, 0, −1, 0, 1, 0, −1,−2, −1, −1, 0, 0, 0, 0, −3, −4, 0, 0, 0, 1, 0, 0, 23, 29, 20, 2, −4, −1,2, 0, 106, 121, 61, −8, −17, −3, 2, 2, 113, 127, 65, −11, −20, −2, 1, 0}, { 0, 0, 0, 0, 0, 1, 1, −1, 0, 0, 0, 0, 0, 1, 3, −3, 0, 0, 0, 0, 0, 1,0, 4, 0, 0, 0, 0, 0, −1, −1, 17, 1, −1, 2, −1, 3, −7, 43, −22, 4, −4, 4,−2, 4, −4, 105, −136, −2, 6, −6, 3, −9, 66, −57, −42, −5, 8, −8, −1, 0,57, −126, 66 }, { 0, −1, 0, 2, 1, −1, −1, 0, −1, −1, 0, 2, 1, −2, −1,−1, 0, −1, 0, 0, 0, −2, −1, −1, −1, −2, −1, −3, −1, −4, −1, −3, −2, −4,−4, −4, −3, −12, −10, 3, −3, −6, 0, 25, 23, −14, −14, 5, 0, 6, 36, 132,104, 13, −16, −5, 1, 9, 43, 139, 110, 5, −12, −4 }, { 0, 0, 0, 0, −1,−1, 4, 9, 0, 0, −1, 0, −2, −4, 10, 30, 0, 0, −1, 0, −3, −7, 24, 80, 0,1, −1, 0, −5, −13, 51, 187, −1, 0, −1, 0, −5, −15, 27, 130, −1, −1, 0,3, −8, −27, −14, 5, 0, −1, 1, 0, −3, −7, −8, −17, 1, −1, 0, −1, 4, −2,−1, −4 }, { 0, −1, 1, −1, 1, −4, 2, 0, 1, −1, 1, −1, 2, −5, 6, 1, 1, −1,1, −1, 1, −4, 1, 0, 1, −1, 1, −1, 2, −6, 4, 2, 2, −4, 4, −4, 11, −31,42, −20, 7, −10, 11, −13, 34, −87, 119, −35, 8, −14, 17, −16, 44, −140,78, 17, 4, −10, 10, −20, 32, −78, 21, 27 }, { 0, 0, −1, 2, 0, −2, 0, 0,0, 0, −1, 2, −1, −2, 2, 1, 0, 0, −1, 1, 1, −3, 0, 2, 0, 0, 0, 0, 1, −3,−3, 1, 1, −1, 2, −5, 12, −19, 10, −3, 3, −2, −1, 4, 18, −37, 37, −3, −7,13, −28, 83, −74, −22, 21, 9, −14, 23, −48, 128, −163, 14, 34, −5 }, {0, 0, 1, 1, 0, −3, −4, −3, 0, 0, 1, 1, 0, −3, −6, −3, 1, 1, 1, 2, 2, −3,−4, −2, 1, 1, 2, 2, 3, −5, −8, −6, 0, 0, 3, 4, 0, −28, −50, −29, −2, −1,1, 5, −19, −74, −120, −67, −2, −2, 2, 4, −23, −101, −110, −55, 0, −2, 0,1, −21, −73, −65, −22 }, { 0, 0, 0, 0, 0, 2, 5, 6, 0, 0, −1, 0, −1, −3,−8, −7, 0, 0, 0, 0, 1, 1, 6, 10, 0, 0, 0, 0, 0, −1, 0, 7, 0, 1, 1, 1, 2,5, 4, −1, −2, −1, −2, −1, −8, −24, −82, −112, 0, 1, −4, −6, 1, 40, 126,138, −1, 0, 2, 11, −1, −12, −65, −65 }, { 0, 0, 0, 0, 0, −3, −1, 1, 0,0, 0, −1, 1, 3, 2, 0, 0, 0, 0, 0, 0, −4, −2, 1, 0, 0, 1, 0, 0, 2, 2, 1,0, −1, 1, −1, 0, −4, 4, −9, 1, −3, 2, 1, 2, −33, 16, 17, −1, −2, 5, 9,−3, −103, −54, 73, 0, 7, −8, −30, 24, 191, −10, −82 }, { 0, 0, 0, 0, 0,1, −4, 3, 0, 0, 0, 0, 0, −2, 7, −5, 0, 0, 0, 0, 0, 1, −6, 7, 0, 0, 0, 0,0, −2, 3, 1, 1, −1, 1, −1, 2, −3, 1, 5, 2, −2, 2, 0, 0, −13, 63, −61,−6, 8, −9, 11, −20, 61, −136, 109, 4, −9, 9, −12, 21, −75, 110, −64 }, {0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0, 0, 0, 0, 0, −2,−3, 0, −1, 0, 0, −1, −2, −6, −12, 0, 0, 0, −1, 1, 4, 18, 24, 1, 2, 1, 1,10, 29, 74, 106, 0, 2, 1, 6, 4, 4, −21, 16, −3, −3, −1, 0, −24, −80,−150, −128 }, { 0, 0, 0, 0, 0, 1, 1, −2, 0, 0, 0, 0, 0, 1, 2, −5, 0, 0,0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 4, −1, 1, 0, 0, 0, 1, 3, 7, −14, 2,−2, 1, −2, 1, 9, 21, −78, 4, −3, 3, −4, 5, 21, 53, −150, 3, −4, 4, −4,10, 19, 64, −168 },

Further, in an embodiment, an 8×8 NSST kernel for the planar mode may bedefined as shown in Table 12 below.

TABLE 12 { −216, 84, 26, −3, −2, 4, 1, 0, 94, −32, −15, 2, 1, −1, −1, 0,25, −12, −3, 2, 0, 0, 0, 0, −12, 7, 3, −1, 0, 0, 0, 0, −4, 1, 1, 0, 1,0, 0, 0, 3, 0, −1, 0, 0, 0, 0, 0, 0, −1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,0, 0, 0, 0 }, { −84, −42, 54, −3, 1, 3, 2, −1, −173, 84, 12, −9, −2, 2,1, 0, 118, −27, −39, 9, 1, −2, −2, 0, 7, −11, 3, 2, 1, 0, 0, 0, −10, 3,8, −3, 0, 0, 0, 0, 2, 1, −2, 0, 0, 0, 0, 0, 5, −1, −1, 0, 0, 0, 0, 0, 0,0, 0, 0, 0, 1, 0, 0 }, { 33, 199, −101, −5, −2, −1, −3, 2, −90, −30, 39,5, −2, 2, 0, 0, 21, −55, 22, 8, −3, 3, 0, 0, 10, 9, −12, −2, 3, −1, 0,0, 0, 7, −2, −2, 0, 0, 0, 0, −2, −3, 1, 1, 0, 0, 0, 0, 1, −2, 1, 0, 0,0, 0, 0, 0, −1, 1, 0, 0, 0, 0, 0 }, { −87, −37, −103, 60, 6, 2, −9, 4,−51, 84, 14, −16, −7, 3, 1, 0, −136, 30, 65, −21, −7, 4, 3, 0, 72, −30,−28, 10, 6, −3, −2, 0, 25, −8, −12, 5, −1, 0, −1, 0, 2, −2, 3, 0, 0, 0,0, 0, −7, 3, 1, −1, 0, 0, 0, 0, 4, 0, −1, 0, 1, −1, 0, 0 }, { 39, 65,33, −6, −16, 7, −1, 1, 56, 184, −118, 15, −2, 0, −5, 3, −6, −71, 31, 12,2, 1, 0, 0, −13, −7, 24, −9, −2, 1, 1, 0, −7, 0, −5, 2, 2, 0, 0, 0, −1,4, 0, −1, 0, 1, −1, 0, −2, −4, 1, 1, 1, −1, 1, 0, 0, 1, 0, 0, −1, 1, 0,0 }, { −1, 68, 150, −43, −11, 8, 7, −2, −88, −47, −61, 27, 5, −2, −3, 1,−127, 53, −15, 6, 3, 1, −1, 0, 35, −11, 5, −6, −4, 0, 0, 0, 30, −14, 0,2, 0, 0, 0, 1, −5, −2, −2, 0, 0, 0, 0, −1, −4, 3, −2, 1, 1, 0, 0, 0, 0,0, −1, 0, −1, 0, 0, 0 }, { −19, −67, 0, −52, 42, −25, 12, −7, −34, −73,−70, 43, −5, 0, −5, 3, 4, −123, 135, −9, −17, 11, 0, −1, −3, 54, −32,−26, 15, −8, 2, −2, 31, 14, −38, 17, −2, 0, 0, 0, −9, −7, 12, −3, −3, 1,0, 0, 7, −6, 2, 0, 3, −2, 0, −1, −1, 0, 2, 0, −2, 1, −1, 1 }, { −8, 6,−64, 34, −14, 9, −9, 4, −36, −20, −109, 54, −5, 0, −7, 4, 54, 120, 29,−47, 3, −2, 4, −1, −105, −39, 53, −14, −3, 4, 0, −1, 73, −7, −22, 16,−5, 1, −1, 0, −26, −7, 6, −1, 0, 0, 0, 0, 14, 10, −3, −2, 2, −1, 0, 0,−11, −3, 1, 0, −1, 0, −1, 1 }, { −10, −1, 18, 25, 4, 2, −1, 1, −11, 37,76, −31, 5, 1, 6, −3, −63, −47, −43, 17, −12, 4, −4, 2, −138, 98, 11,−12, 5, −2, 1, 1, 114, −56, −25, 16, −1, −1, −1, 0, −36, 11, 7, −2, −1,1, 0, 0, 9, −7, −4, 1, 0, 0, 0, 0, −10, 8, 1, 0, 0, 0, 0, 0 }, { 11, −1,45, 135, −114, 74, −28, 17, −13, −44, −22, −59, 59, −34, 13, −8, 7, −55,41, −56, 47, −26, 12, −6, −1, 13, −15, 32, −32, 15, −6, 3, −13, 20, −8,8, −3, 2, −1, 1, 0, −2, −1, −3, 3, −1, 0, 0, 2, −5, 4, −1, 1, 0, 0, 0,1, −1, 0, −2, 0, 0, 0, −1 }, { 18, 30, 70, −26, 27, −9, 10, −5, 27, 30,70, −69, −8, −3, 8, −6, 37, 36, 92, −19, −16, 9, 3, 0, −20, −95, −72,54, 21, −9, −3, 2, 77, 68, −33, −11, 2, −2, 0, 1, −48, −33, 13, 4, 0, 2,0, 0, 26, 10, 2, −1, −2, 0, 1, 0, −8, −10, 0, 1, 0, 0, 1, −1 }, { −13,−26, −34, −11, 6, −14, 4, 0, −13, −37, −26, −51, 24, −5, 0, −4, −43,−75, −37, 85, 10, −3, −5, 2, −23, −103, 109, 14, −55, 14, 2, 1, 31, 73,−16, −74, 34, −7, 2, −3, −25, −10, −19, 13, 11, 6, −4, 2, 5, −8, 11, 8,−12, −3, 2, 1, −4, −3, 2, −1, 2, −2, 0, −3 }, { 15, 0, 5, 18, 4, −1, 0,0, 35, −12, 15, 33, 11, −2, 4, 1, 50, −52, −40, −39, 15, −12, 1, −3,129, −60, 43, −21, 5, 4, 0, −1, 140, −95, −13, 33, −9, 1, −1, 1, −18,14, 9, −6, 1, −4, 1, −1, 6, −1, −7, −5, 6, 0, −1, 0, −1, 4, 2, 0, 0, 1,−1, 2 }, { 16, 24, 38, 150, 151, −49, 9, −3, −6, −27, −50, −55, −35, 14,−1, 1, 6, −13, −17, −14, −66, 23, −5, 6, 2, −3, 8, 5, 17, −10, 0, −3,−23, −13, 25, −3, 12, −2, 0, −2, 4, 6, −5, −6, −3, 1, 0, 1, 0, −2, −1,2, −3, 0, 0, 0, −1, 1, −2, 0, 1, −1, 1, −1 }, { −2, 4, 24, 20, 35, −13,3, −4, −1, 13, 65, 103, −34, 37, −9, 8, −38, −46, −37, −120, 37, −33,11, −8, −30, −33, 59, 13, 0, 7, −2, 0, −47, 81, −75, 38, −13, 2, −4, 2,9, −28, 20, −4, 0, −2, 2, −1, −3, 1, −6, −7, 3, 0, 0, −1, 5, −8, 9, 1,−1, 2, −1, 0 }, { −3, 11, 7, −74, 23, −31, 14, −9, −9, 11, 26, −111, 29,−33, 15, −10, 0, 23, 71, −106, −9, −5, 9, −6, 0, 23, 126, −21, −48, 22,0, −1, −34, −48, 21, 17, 0, 3, 1, −1, 27, 28, −1, −14, 5, −2, 1, 0, −6,−9, −1, −1, 2, −4, 1, 0, 3, 4, 4, −1, 1, −1, −1, 0 }, { 5, 11, 41, 66,40, 4, 11, 1, 9, 24, 73, 95, 68, 10, 3, 3, 26, 54, 94, 76, 26, 10, 1, 2,33, 55, 58, −47, −63, 14, 3, 1, 13, 30, −1, −50, 17, −4, 4, −2, 3, 24,−8, 9, 8, 6, −3, 2, 4, −7, 14, 10, −4, −2, 2, 1, 1, 5, 0, −1, 0, −1, −1,−1 }, { 3, 5, −19, −17, −9, −2, −2, 1, 8, 13, −43, −38, −21, 8, −4, −1,20, 35, −60, −51, −40, 0, −2, 0, 83, 134, 10, 25, −23, −4, 2, 2, 68,125, −45, −40, 24, −6, 1, −1, −17, −3, 7, 1, 1, 8, −2, 1, 8, −6, 3, 6,−11, −2, 1, 0, 3, 5, 4, −1, 0, −1, 1, −1 }, { 3, 5, 7, 14, 6, −11, 0, 1,−1, −8, −10, −81, −126, 60, −17, 3, 12, 23, 24, 72, 130, −61, 19, −5,17, 20, 11, −59, −15, 10, −1, −1, 4, 2, −53, 49, −20, 12, −4, 3, 10, −3,20, −7, −2, −5, 2, −1, −5, 5, −5, 1, 9, 0, 0, −1, 5, −3, 2, −3, −4, 1,−1, 1 }, { −4, −1, −6, −1, −5, 1, −3, 0, 0, −3, −3, 3, 5, 0, −1, 0, −8,6, −7, 10, −5, 4, 0, 1, 19, −5, 2, −3, −10, 7, −2, 0, −66, −10, −15, 20,15, −8, 2, −1, −149, 75, 79, −47, −19, 9, −2, −1, 110, −50, −60, 32, 11,−6, 0, 1, −44, 16, 23, −10, −5, 4, 1, −1 }, { −3, −3, −4, 5, 13, −9, 7,3, 1, 6, 0, −5, 8, −22, 9, −3, −14, −14, −10, −25, 35, −3, 1, −4, 3, 12,4, −89, 22, 33, −8, −2, 20, 63, 143, −16, −105, 7, 6, 2, −61, −70, 16,51, −6, −29, 5, −4, 25, 27, −29, −23, 28, 11, −7, −3, −12, 1, 3, −10,−1, 5, 2, 4 }, { −2, −1, −3, −14, 22, −25, 12, −8, −1, 2, −10, 31, −54,45, −22, 12, −3, −13, 10, −57, 75, −49, 21, −12, −16, 13, −46, 93, −68,32, −15, 9, 19, −43, 85, −84, 34, −15, 9, −7, −32, 47, −56, 30, −4, 7,−5, 4, 16, −29, 28, −10, −2, −3, 3, −2, −14, 16, −14, 7, −1, 0, −1, 0 },{ −2, 6, −24, −16, 86, −52, 20, −10, −2, 11, −38, −7, 119, −68, 26, −8,1, 16, −45, 24, 74, −30, 9, −3, −1, 12, −63, 52, −54, 49, −23, 10, 0, 6,−54, 69, −45, 14, −8, 4, 6, −4, −1, 6, 7, −11, 4, −2, 1, −3, 4, −14, 16,−1, −1, 0, 5, −3, 1, −2, 1, 0, −1, 1 }, { 1, −2, −1, −32, 79, 121, −118,23, 1, 1, 2, 5, −11, −26, 20, 0, −1, −1, 2, 16, −53, −96, 97, −20, −1,0, 3, −9, 9, 36, −37, 2, 1, −2, 3, −18, 9, 34, −35, 8, 1, 3, 1, −2, −2,−8, 11, −1, 0, 2, −1, 2, 2, −7, 5, −3, −1, −2, 3, 3, −4, −4, 2, 0 }, {0, 1, 2, 1, −7, 8, −1, 1, 0, −3, 4, −8, −2, −6, 3, −2, −3, 6, −1, −2,−10, 4, −3, 0, 5, −11, 11, −35, 59, −25, 5, −2, −9, 38, −25, 45, −56,28, −7, 5, −42, 102, −134, 49, 24, −20, 9, −2, 30, −75, 94, −41, −5, 7,−3, −1, 4, 5, −14, 10, −4, 3, −5, 3 }, { −1, 1, 5, 3, −46, −97, −29, 19,0, −1, −7, −11, 81, 149, 27, −26, 1, 5, 8, 7, −66, −120, −11, 17, −2,−3, −7, −9, 21, 19, −2, −1, −2, −4, 3, 2, 0, 36, 3, −8, 0, −6, 5, 9,−10, −22, 2, 6, 1, 3, −3, −8, 5, 6, −1, −3, −1, 0, 2, 1, 1, −3, 0, 1 },{ 2, 1, 0, 1, −3, 0, 0, 0, 2, 2, 1, −1, −4, −2, 2, 0, −5, 2, 1, 0, 0,−1, 0, −1, 0, −1, −4, −4, 5, −3, −1, 1, 14, −4, −11, 1, 7, −2, −1, 2,119, −53, −72, 41, 5, −4, −2, 0, 146, −64, −83, 47, 12, −8, −4, 2, −62,17, 41, −21, −12, 10, 0, 1 }, { −1, −3, −1, 2, −1, 0, −5, 1, −3, −4, 3,14, −1, −1, 4, 1, 0, 4, 30, 55, 28, 0, 7, 2, 16, 38, 92, 145, 106, 15,3, 5, 15, 27, 76, 96, 34, 10, 5, 4, −4, −13, 4, 0, −10, −2, 4, −1, 5, 4,−9, −5, 6, 6, −3, −1, 1, 4, 4, 5, 3, 2, 1, 1 }, { −1, 4, −6, −3, 30,−20, 5, −5, −3, 6, −10, −13, 63, −41, 12, −9, −4, 7, −12, −32, 97, −62,23, −9, −5, 6, −7, −53, 124, −93, 32, −12, −2, 3, 0, −66, 87, −28, 11,−9, −4, 2, 5, −18, −4, 18, −2, 1, −5, −6, 6, 21, −20, −2, 5, 1, −4, −1,3, 3, −7, 2, −1, 0 }, { 0, 0, 1, 1, 1, 4, −3, −2, 2, 3, 1, −1, −2, 2,−2, −1, 10, 13, 7, 2, −7, −2, 0, 1, 20, 25, 18, 11, −11, −17, 1, 1, −44,−64, −43, 4, 23, 9, −4, 2, −115, −160, −65, 52, 40, 10, −4, 2, −46, −60,−3, 27, 0, −8, 0, 1, 18, 11, −1, −12, −12, 0, 0, −1 }, { 0, −2, −14,−24, 53, 123, 32, −23, 0, −3, −17, −28, 64, 140, 28, −20, −1, −2, −8,−11, 29, 46, 1, −2, −1, −1, 5, 14, −21, −81, −15, 18, −1, −2, 11, 25,−31, −73, −8, 10, 2, −3, 6, 16, −18, −3, −1, −6, 3, 3, −1, −2, −4, 15,−5, 1, 1, 0, −1, −6, −1, 11, 0, 0 }, { 1, −1, −9, −2, 30, 93, 133, 54,0, 1, 0, 0, −9, −22, −26, −21, −1, 2, 5, 1, −31, −102, −116, −36, −1,−1, −2, 0, 10, 37, 47, 22, −2, −3, −4, −1, 15, 35, 37, 12, −1, 1, 0, −1,−2, −11, −11, −6, 1, 2, −4, −2, 3, 0, −1, 1, 0, 0, 1, 2, 4, −2, −1, 0 },{ 1, 2, 4, 4, −2, 0, 3, 1, −2, −5, −7, −7, −5, −4, 4, 0, 5, 7, 12, 9, 7,4, −4, 0, 7, 16, 35, 61, 76, 25, −2, 3, −22, −50, −79, −115, −106, −22,−3, −2, 14, 27, 70, 97, 26, −3, 6, −2, −1, −7, −31, −43, −3, 4, −6, −1,2, 5, 12, 9, 6, 5, 1, 3 }, { 0, 1, 3, 1, −8, −2, −2, 0, 0, 1, 2, −6, −5,−13, 9, 2, 1, 4, −3, −1, −14, 6, 0, −3, 2, −6, −10, −45, 17, 96, −39, 6,−10, −1, −25, 35, 97, −123, 35, 1, −4, −3, 2, 89, −85, −26, 27, −9, 0,−2, 15, −46, −8, 59, −27, −1, 2, 3, −4, −12, 19, −2, 2, 9 }, { 0, 0, 0,1, 0, 0, 2, 1, 0, 1, 1, 2, 3, 2, 1, −2, 2, 4, 8, 10, 9, 7, 1, 1, 4, 9,13, 13, 21, 20, −4, 2, −3, −7, −27, −50, −44, −19, 0, −4, −13, −33, −88,−143, −98, −42, −1, −5, −7, −13, −64, −105, −38, −1, −6, −1, 1, 9, −12,−19, −3, −2, −1, 1 }, { 0, −1, 3, −13, 24, 2, −92, 73, 0, 0, −4, 17,−42, −13, 140, −90, 0, −1, 4, −10, 37, 20, −107, 55, −1, 1, 0, 7, −14,−6, 6, 3, 0, −1, 0, −4, 1, −8, 50, −27, −1, 4, −5, −5, 11, 4, −22, 15,0, −2, 3, 3, −6, 2, 5, −3, −2, 1, −2, 0, 4, −1, −4, 0 }, { 0, 0, 0, 0,3, 4, −2, −2, 0, 0, 0, 1, −3, −12, 1, 4, 0, −1, −1, −2, 14, 34, 4, −10,0, 2, 6, 11, −33, −85, −12, 15, −4, −8, −13, −15, 53, 133, 16, −21, 0,5, 27, 55, −72, −116, 5, 17, 1, 1, −21, −45, 46, 67, −10, −15, −1, 1, 6,13, −15, −26, 2, 11 }, { −1, 0, 2, 1, 0, 0, 1, 0, 0, 0, 1, 0, −2, 1, −1,−1, 2, 0, 1, 1, −2, 3, −2, −3, 5, −1, −4, 3, 1, 2, −3, 2, −2, 4, −3, 4,0, −1, 5, 2, −7, 12, −10, −4, 12, −9, 8, −2, −87, 24, 33, −18, 1, 0, 1,−2, −169, 30, 140, −59, −43, 21, −1, 1 }, { 1, 1, 1, 0, 1, 0, 1, 0, −1,−2, 0, 1, −1, 1, 3, 0, 4, 5, 3, −3, 0, 3, −3, −1, −16, −24, −13, −4, 13,18, −4, 1, 28, 33, 21, −4, −19, −7, 4, 2, 23, 36, 25, −9, −33, −20, −3,−2, −101, −140, −47, 73, 47, 6, −6, −3, 76, 85, 21, −56, −27, 12, 8, 2}, { 0, 0, 4, −4, −6, −47, −8, 166, 0, 0, 1, 1, 4, 10, −12, −31, 1, 0,−3, 3, 5, 44, 34, −145, 0, 0, 1, 1, −4, −24, −4, 64, −1, −1, 0, 1, −1,−18, −27, 57, 0, −1, 0, 1, 0, 8, 4, −30, 0, 0, 0, −2, 0, 0, 4, −2, 3, 0,−2, 0, −2, 5, 1, 2 }, { 0, 0, 3, −17, 32, 10, −95, 73, 0, 1, 3, −18, 38,14, −116, 81, 0, 0, 0, −5, 13, 8, −31, 15, 0, −1, −5, 12, −19, −6, 105,−68, 1, −1, −1, 17, −22, −23, 77, −39, 0, −1, −1, 12, −9, −9, 0, 8, 1,0, −2, −4, −1, 12, −16, 6, 2, 0, 3, −4, 2, 8, −12, 6 }, { 0, 0, 1, −1,1, −4, 1, 1, 0, −1, 4, −2, 4, −4, 3, 0, 3, −6, 6, −5, 1, −5, 4, 0, 2,−3, 1, −1, 3, −5, 4, −3, −7, 16, −25, 17, −5, 5, −9, 2, −29, 59, −81,56, −22, 22, −14, 6, −56, 108, −126, 66, −10, 2, −1, 2, −41, 75, −74,27, 2, −5, 3, 1 }, { 0, 0, 0, 0, 2, −2, 1, 1, 0, 0, −1, −1, 4, −4, 0, 0,0, 0, −1, −3, 7, 5, −9, 2, 1, −1, 4, 2, −4, 11, −8, 1, 2, −8, 7, 3, −42,58, −3, 4, 4, −15, 18, 28, −126, 77, 6, −1, 4, −20, 39, 63, −145, 35, 0,1, 0, −17, 23, 46, −71, 22, −11, −3 }, { 0, 0, 0, −3, 3, −2, 1, 0, 0, 1,−2, 5, −4, 3, −1, 1, 0, −2, 3, −5, 11, 0, −1, −3, −2, 2, −4, 6, 2, 8,−2, −3, 2, −3, 7, −3, −12, 6, −11, 13, −3, 4, 2, −30, 80, −84, 52, −17,1, 0, −6, 63, −126, 107, −49, 11, −4, 6, 2, −51, 82, −59, 26, −7 }, { 0,−1, 0, 1, −1, −3, 1, −2, 0, 1, 0, 2, −2, 1, 0, −1, 1, −3, 3, −3, 2, 0,−3, 1, −3, 5, −5, 4, 0, 3, −9, 3, 3, −2, 2, −4, 8, −8, 4, −2, −27, 48,−60, 19, 7, −13, 11, −4, −11, 47, −72, 31, 5, −5, −2, 2, 98, −150, 121,−32, −27, 24, −8, 1 }, { 0, 0, 0, 1, −4, 2, 9, −4, 0, −1, 1, −3, 6, 2,−22, 19, 0, 0, −2, 5, −12, 0, 35, −40, 1, −1, 4, −9, 19, −3, −99, 72, 0,2, −6, 10, −22, 0, 140, −92, 3, −1, −3, −17, 35, 2, −85, 60, −1, −1, −1,18, −34, 8, 40, −32, 0, 0, −3, −8, 16, −1, −19, 14 }, { 0, −1, −1, −2,1, 6, 54, 85, 0, 1, 3, 5, 2, −8, −92, −144, 0, −2, −3, −3, −1, 10, 81,122, 1, 0, 0, 2, 0, 0, −23, −26, 0, 0, 0, 4, −1, −3, −12, −41, 1, 1, 1,3, 0, 1, 17, 28, 0, −2, 0, 1, 1, 2, −8, −10, 0, 0, 1, −2, −4, −5, 2, 2}, { 1, 1, 0, −1, 1, 0, −1, −1, 0, −1, −1, 0, −1, −1, 1, 1, −2, −2, 0,0, −1, −3, −2, 0, −3, −4, −1, 2, −4, −10, −3, −1, −8, −9, −5, 2, 15, 14,4, 1, −5, −3, −3, 8, 22, 22, 6, 0, 68, 90, 30, −44, −33, −8, −2, −2,107, 144, 54, −87, −66, −1, 2, −2 }, { 0, 1, 5, 8, −13, −33, −14, −3, 0,1, 9, 15, −27, −70, −26, 5, 1, 2, 14, 25, −40, −106, −38, 11, 2, 3, 19,28, −55, −137, −50, 13, 2, 3, 17, 25, −51, −100, −27, 2, 1, 0, 11, 16,−31, −21, 0, −2, 0, −3, 2, −5, −10, 12, −5, −1, 1, 0, 0, −8, −2, 8, 2, 0}, { 0, 0, −1, −1, −1, −3, 2, 2, 0, 0, 1, 1, −1, −2, 2, 0, −1, −2, −1,−2, 0, 4, 0, −1, 6, 11, 18, 22, 27, 22, −1, −1, −10, −20, −30, −36, −38,−17, 4, 2, −4, −11, −20, −32, −57, −45, −12, −3, 32, 59, 92, 120, 90,35, 1, −4, −16, −32, −61, −87, −51, −4, 8, 2 }, { 0, 1, 2, 3, 5, −5,−77, −110, 0, 1, 2, 3, 5, −7, −87, −126, 0, 1, 0, 0, 1, 1, −22, −27, 0,−1, −2, −3, −3, 10, 69, 109, 0, 0, 0, 0, −4, 1, 38, 63, 1, 1, 0, 2, −2,3, −1, −8, 1, 2, −3, −3, 3, 10, 1, −9, 1, 1, −3, −1, 2, 4, −3, −6 }, {0, 0, 0, 0, −1, −1, 1, 1, 0, −1, −1, −2, −2, −1, 0, −2, 0, 1, 1, 0, 2,2, 3, 2, 1, 1, 1, 2, 6, 9, 4, 5, −5, −8, −12, −16, −19, −19, −8, −8,−10, −19, −27, −31, −33, −34, −14, −3, 16, 33, 45, 59, 48, 25, 12, 3,34, 72, 109, 137, 103, 41, 13, 4 }, { 0, 0, 1, −10, 15, 7, −46, 30, 0,0, 3, −16, 28, 14, −86, 53, 1, 0, 3, −19, 36, 22, −109, 67, 1, 1, 2,−18, 36, 21, −120, 67, 0, 1, 0, −15, 29, 23, −83, 38, 1, 2, −2, −9, 13,5, −22, 12, −1, 2, 0, 2, 7, −9, 2, 0, −3, 1, −2, 2, 1, −5, 10, −3 }, {0, 0, 0, 0, 0, 2, −2, −3, 0, 0, 0, −1, −1, 0, 11, 14, 0, 0, 1, 0, 0, 1,−18, −31, 0, −1, −2, −3, 0, 14, 64, 85, 0, 2, 4, 5, −2, −20, −94, −135,−1, −3, −3, −5, −2, 18, 73, 107, 1, 0, 3, 4, 6, −2, −46, −65, 0, −2, 1,1, −11, −8, 17, 27 }, { 0, 1, 0, −1, 0, 0, −2, 0, 0, 0, 0, 0, 0, 0, −2,0, 0, −1, 0, −2, 4, 5, −8, 4, 0, 0, 3, 2, 1, −18, −23, 30, −1, −2, −3,−5, 0, 43, 18, −41, −2, 0, 3, 17, −46, 10, 100, −45, −2, 8, 12, 17, −5,−84, 4, 50, −3, 7, −16, −74, 116, 64, −109, 8 }, { 0, 0, 0, 0, 0, 2, −1,−2, 0, 0, 0, −2, 2, 3, −1, 4, 0, 0, 1, 2, −1, −7, −11, −6, 0, 0, 5, 3,2, −20, −59, −13, 1, 1, −4, −14, −18, 9, 81, 36, 1, −2, −18, −31, 4, 95,113, −6, −2, 1, −4, 4, 67, 24, −89, −26, 0, 5, 20, 39, −15, −109, −17,40 }, { 0, 0, 1, −1, 1, 0, −1, 2, 0, 0, 0, 0, 1, −1, 2, 0, 0, 0, 0, −1,3, −3, 8, −2, 1, −1, 2, 0, −2, −15, 25, −9, 0, −2, 0, 5, −15, 24, −23,10, −2, 4, −8, 33, −73, 85, −80, 32, −1, −4, 10, −15, 15, −9, 12, −8, 2,−15, 38, −85, 123, −106, 71, −28 }, { 0, −1, −1, −1, −3, −4, 27, 44, 0,−1, 0, −1, −6, −8, 48, 87, 0, −2, −1, −2, −8, −11, 65, 121, −1, −2, −1,−2, −8, −10, 69, 137, −1, −1, 0, −1, −6, −5, 45, 83, 0, −1, 0, −1, −1,2, 15, 20, 0, −1, −2, −2, −1, 12, 7, 0, 1, −1, −1, −3, −1, 8, −1, −4 },{ 0, 0, 0, 0, 1, 2, −4, 2, 0, 0, 0, −1, 3, 4, −8, 6, 1, 0, 0, −1, 5, 6,−14, 6, 1, 0, 0, 0, 1, 0, −9, 5, 0, −1, −2, 4, −4, −10, 34, −21, 0, −1,−2, 10, −12, −33, 108, −56, 0, 0, −1, 14, −16, −60, 149, −68, −1, 2, −1,14, −14, −48, 108, −44 }, { 0, 0, −1, −1, 1, 1, −1, 0, 0, 0, 0, 0, 0,−3, −3, 2, 0, 0, 2, 2, −4, −10, −6, 1, 0, 1, 2, 0, −6, −9, −9, −7, 1, 0,−2, −6, 2, 27, −4, −22, 2, −1, −10, −19, 24, 101, 35, −26, 1, −5, −21,−36, 45, 157, 64, −11, −1, −8, −20, −34, 26, 116, 45, −7 }, { 0, 0, 0,0, 0, −1, 2, 1, 0, 0, 0, 0, 0, 0, 1, −4, 0, 0, 0, 1, 2, 3, −8, −26, 1,1, 0, 1, 1, −1, −21, −52, 0, −2, −2, −3, −1, 0, 22, 81, −1, −1, 0, −2,−3, −14, 49, 157, 1, 2, 0, 3, 0, −8, 0, −44, 1, 2, −6, −5, 17, 48, −29,−148 }, { 0, 0, 0, 0, 0, 1, 1, 2, 0, 0, 0, −1, 0, 0, −2, 1, 0, 0, 0, 0,−1, −2, −3, −1, 0, 0, −2, −1, 4, 11, 19, 19, 1, 2, 4, 2, −6, −24, −39,−39, 0, −1, −4, −3, 6, 11, 5, 1, −1, −5, −9, −5, 23, 71, 106, 80, 3, 8,19, 18, −30, −108, −135, −76 }, { 0, 0, 0, 0, 0, 0, −2, 2, 0, 0, 0, 0,−1, 0, −2, 1, 0, 0, 0, 0, −1, −1, 7, −5, 0, 0, 1, −3, 2, −5, −15, 45, 1,0, −2, 1, −2, 8, 29, −67, 1, 1, −2, −2, 0, 9, 10, −29, −2, 1, 3, −6, 17,−17, −85, 140, −2, −2, −4, 19, −28, 11, 98, −135 }, { 0, 0, 0, 0, 0, 0,1, 4, 0, 0, 0, 0, 0, −1, 4, 10, 0, 0, 0, 0, −1, −2, 7, 19, 0, 0, −1, −1,0, 1, 7, 16, 0, 0, −1, 0, 3, 4, −12, −37, 1, 0, −1, 1, 4, 8, −47, −121,0, 1, 0, 0, 6, 7, −69, −154, −1, 1, 3,2,5,1, −61,−118 },

The transform kernels of Tables 9 to 12 described above may be definedas smaller transform kernels for the reduced transform.

For example, for the DC mode and the 8×8 NSST with R=8, the memory andthe multiplication may be reduced by ½. Accordingly, the reducedtransform kernel may be defined with a smaller size as shown in Table 13below by maintaining only coefficients (8×16 matrix) of an upper half ofthe transform kernel of Table 7 above.

TABLE 13 { 223, −84, −36, 3, −79, 26, 16, −2, −18, 8, 5, −1, −2, 0, −1,1 }, { 76, 216, −82, −32, 8, −51, 10, 8, −27, −39, 17, 10, 2, 3, −1, −1}, { −54, 39, 20, −8, −215, −6, 71, 2, 88, −14, −31, 2, 15, 2, −7, −1 },{ 6, −47, −4, 13, 7, −229, 39, 50, 1, 69, −9, −20, −3, 35, −2, −12 }, {58, 40, 203, −108, 4, −19, −51, 26, −6, −1, −56, 21, −7, −2, 13, −3 }, {36, 8, −33, 0, 65, 9, −9, −2, 196, 10, −64, −3, −120, 5, 42, 0 }, { 18,27, 47, 22, 65, 37, 206, −50, −27, 60, −62, −9, −3, −34, −37, 16 }, { 1,51, −10, 26, −32, 30, −73, 16, −4, 194, −8, −72, 19, −98, 32, 28 },

The transform kernel of Tables 9 to 12 described above may be defined asa smaller transform kernel for the reduced transform.

For example, for the DC mode and the 4×4 NSST with R=8, the memory andthe multiplication may be reduced by ½. Accordingly, the reducedtransform kernel may be defined with a smaller size as shown in Table 13below by maintaining only coefficients (8×16 matrix) of an upper half ofthe transform kernel of Table 7 above.

TABLE 14 { 218, −83, −33, 0, −5, −1, −2, 0, −89, 37, 13, 0, 2, 0, 1, 0,−18, 8, 3, −1, 1, 0, 0, 0, −4, 1, 0, 1, 0, 0, 0, 0, −4, 2, 1, 0, 0, 0,0, 0, −2, 0, 0, 0, 0, 0, 0, 0, −2, 1, 0, 0, 0, 0, 0, 0, −1, 0, 0, 0, 0,0, 0, 0 }, { 77, 188, −115, −21, −12, −3, −5, −1, 40, −65, 23, 6, 0, 1,0, 0, −47, −27, 27, 7, 2, 1, 1, 1, 4, 1, −3, −1, 1, 0, 0, 0, −8, −5, 5,1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, −3, −2, 2, 0, 0, 0, 0, 0, 0, 0,0, 0, 0, 0, 0, 0 }, { 45, −76, −3, 20, −2, 4, −1, 1, 197, −6, −68, −4,−6, −2, −3, 0, −102, 46, 29, −8, 3, −1, 1, 0, −8, −7, 8, 2, 0, 0, 0, 0,−14, 5, 1, 0, 0, 0, 0, 0, −3, −1, 2, 0, 0, 0, 0, 0, −5, 2, 1, 0, 0, 0,0, 0, −1, 0, 1, 0, 0, 0, 0, 0 }, { 70, 75, 181, −106, −6, −17, 2, −6,20, −1, −72, 31, 3, 3, −1, 1, 7, −34, −39, 22, 3, 3, 0, 1, −16, 8, 11,−5, 0, 0, 0, 0, −2, −5, −6, 5, 1, 1, 0, 0, −3, 1, 2, −1, 0, 0, 0, 0, −1,−2, −2, 2, 0, 0, 0, 0, −1, 0, 0, 0, 0, 0, 0, 0 }, { −17, 39, −34, −8, 4,−2, 1, 0, 11, 212, −53, −45, −5, −8, −1, −3, −17, −91, 52, 22, −1, 3, 1,1, 3, −22, 0, 7, 1, 1, 0, 0, 2, −6, 1, −1, 0, 0, 0, 0, 0, −6, 1, 2, 0,1, 0, 0, 0, −2, 1, 0, 0, 0, 0, 0, 0, −2, 0, 1, 0, 0, 0, 0 }, { 38, −6,−58, 8, 2, 2, 0, 1, 83, 8, −25, −4, −5, 0, −2, 0, 191, −14, −46, −2, −7,−1, −3, 0, −108, 20, 38, −3, 3, 0, 2, 0, 6, −7, 2, 2, 0, 0, 0, 0, −19,4, 5, 0, 1, 0, 0, 0, 3, −2, 1, 0, 0, 0, 0, 0, −6, 1, 1, 0, 0, 0, 0, 0 },{ 21, 6, 54, −2, −11, 0, −1, 0, 75, 47, 203, −71, −25, −7, −2, −2, −12,−11, −70, 32, 11, 2, 1, 1, −7, −7, −28, 8, 6, 1, 0, 1, −7, 1, 0, 0, 0,1, 0, 0, −1, −2, −7, 3, 2, 0, 0, 0, −2, 0, −1, 0, 0, 0, 0, 0, −1, −1,−2, 1, 1, 0, 0, 0 }, { 25, 82, 58, 165, −84, 19, −15, 4, −17, 36, −36,−38, 30, −7, 4, −1, 5, 78, −47, −62, 17, −5, 3, −2, 1, −39, 24, 13, −7,3, 0, 1, −3, −5, −4, −1, 4, 0, 1, 0, 1, −5, 4, 1, 0, 0, 0, 0, −1, −1,−2, −2, 1, −1, 0, 0, 0, −2, 1, 0, 0, 0, 0, 0 }, { 13, −31, 35, 112, −40,13, −8, 3, 9, −66, −8, −13, 17, 0, 2, 0, −11, −187, 20, 23, 5, 8, 1, 2,2, 80, −24, −18, −1, −3, 0, −1, 1, 6, 2, −9, 1, −1, 0, 0, 0, 10, −2, 0,−1, 0, 0, 0, 0, 0, 0, −2, 1, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0 }, { 17,−5, −23, −4, 1, 1, −1, 0, 34, 10, −33, 11, 1, −1, 0, 0, 74, −4, −67, 5,3, 0, −1, 1, 176, 3, −65, −7, 1, −2, −2, 0, −110, 23, 51, −8, 2, 0, 2,0, 34, −6, −8, 2, −1, 0, −1, 1, −19, 3, 7, −1, 0, 0, 0, 0, 6, −2, −1, 0,0, 0, 0, 0 }, { 12, 14, 33, 39, 14, −4, 0, −1, 26, 46, 60, 174, −64, 15,−9, 3, 28, −9, 78, −106, 16, −1, 3, −1, −5, 3, −51, −8, 10, 1, 2, 0, −6,−5, 5, 8, −1, 1, 1, 0, −1, 1, −8, −6, 3, −1, 0, 0, 0, −1, 1, 1, 0, 1, 0,0, 0, 0, −1, −2, 1, 0, 0, 0 }, { 22, 6, 53, −12, −13, 1, 0, 0, 9, −37,14, −99, 23, −5, 4, −1, 79, 32, 179, −3, −47, −1, −6, −1, 14, −26, −62,53, 8, 1, 0, 1, −24, 5, −10, −7, 7, 0, 0, 0, 5, −3, −5, 7, −2, 1, 0, 0,−3, 1, −2, −1, 1, 0, 0, 0, 1, −1, −1, 3, −1, 0, 0, 0 }, { 21, 31, 27,80, 205, −58, 11, −6, −8, −14, −20, −51, −69, 23, −2, 1, −6, −2, −18,−9, −41, 12, −3, 0, 5, 2, 9, 10, 16, −5, 1, 0, −1, −1, 0, −7, −8, 3, 1,0, 1, 1, 1, 4, 3, −1, 0, 0, 0, −1, 0, −2, −4, 1, 0, 0, 0, 1, 0, 1, 2, 0,0, 0 }, { −5, 18, −4, −8, 0, −3, 1, −1, −8, 34, −19, −30, 16, −5, 1, 0,−16, 71, −8, −9, −2, −3, 1, −1, −27, 176, −103, −23, 19, −9, 3, −3, 22,−93, 57, 10, −7, 5, −2, 2, −7, 28, −14, 0, 1, −2, 1, 0, 3, −12, 6, 0, 1,0, 0, 0, −2, 4, −2, 1, −1, 0, 0, 0 }, { 6, 5, 7, 15, 80, −8, 2, −3, 21,22, 38, 54, 215, −39, 5, −3, −1, −2, −1, −5, −48, 25, −2, 1, −2, −4, 1,−10, −49, 11, 3, 0, −1, 2, −1, 8, −3, −3, 1, 1, 0, −1, 0, −3, −9, 1, 0,0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, −3, 1, 0, 0 }, { −4, −16, −14,−64, 7, −6, 4, −1, 1, −17, −6, −77, 14, −8, 5, 0, −16, −60, −40, −201,22, −6, 4, 0, 0, 6, −9, 54, −41, 10, −3, −1, 6, 10, −2, 53, −2, −4, 1,−1, −1, 0, −2, 0, 3, −1, −1, 0, 2, 0, 1, 5, −1, 0, −1, 0, −1, 0, 0, 0,−1, −1, 0, 0 }, { 1, 1, 11, −5, −4, 2, 0, 1, 7, 14, 37, −8, −16, 2, 0,1, 13, 14, 48, −14, −12, 0, 0, 0, 68, 96, 168, −24, −39, −1, −4, −1,−63, −71, −65, 40, 14, 0, 3, 0, 19, 16, 10, −10, −1, 0, 0, 0, −8, −8,−5, 5, 0, 1, 0, 0, 3, 3, 1, −1, 0, 0, 0, 0 }, { 21, 6, −6, 0, −1, 1, −1,0, 32, 9, −3, −2, −2, −1, 0, 0, 37, 14, −3, 0, 2, −2, −1, 0, 116, 33,−13, −13, −4, 1, −1, 1, 191, 42, −88, −6, 2, −2, −2, 1, −33, 2, 18, −1,0, 0, 1, 0, 12, 2, −4, 2, 0, 0, 0, 0, −6, 2, 3, 0, 0, 0, 0, 0 }, { 7,11, 11, −3, 57, 206, −13, −10, −2, 0, −2, 3, −11, −33, 4, 5, −4, −6, −6,2, −29, −128, −6, 9, −1, 1, − 1, 4, 8, 4, −11, −3, −1, 1, 0, 14, −2, 1,1, −2, 0, 0, 1, −1, 3, 1, 0, 0, 0, 1, −1, 0, 0, −2, 1, 0, 0, 0, 1, 0, 0,0, 0, 1 }, { 3, 3, 4, 8, 45, 1, −3, 0, 1, −2, 0, 4, 24, 27, −4, −1, 15,11, 26, 41, 196, −38, 3, −1, 1, 21, 6, 92, −81, −23, −4, 1, −2, −2, 5,−38, −43, 2, −2, 0, 0, 4, −6, 3, 6, −1, 2, 0, 1, −2, 2, −3, −4, 1, 1, 1,1, 1, 0, 1, 1, 0, 1, 1 }, { 9, −18, 3, 7, −4, 2, 0, 1, 13, −25, 1, 8,−3, 3, −2, 0, 14, −33, −2, 13, −9, 4, 0, 0, 46, −100, 13, 24, −11, 2,−3, 1, 69, −187, 81, 38, −25, 10, −5, 3, −9, 28, −17, −6, 5, −1, 0, −1,3, −9, 2, −1, 1, 1, 0, 0, −3, 5, −5, 1, −1, 0, 0, 0 }, { 4, 3, 7, 10,19, −7, 0, 0, −3, −9, −7, −36, −9, −2, 0, −2, 14, 11, 29, 19, 72, −13,5, 1, −13, −39, −41, −165, −65, 39, 2, 2, −4, 21, 2, 139, −3, −14, 3,−4, 1, 0, 4, −17, 11, 2, −1, 1, 0, 0, −2, 4, −3, 1, −1, 0, −1, −1, −1,0, 1, −1, 0, 0 }, { −2, −3, −4, 3, −22, −63, 6, 11, 6, 9, 8, 1, 45, 175,−9, −15, −5, −7, −6, 1, −44, −148, 6, 15, 0, −3, −1, −4, 10, −36, −16,2, 0, 7, −1, 27, 0, 24, −3, −10, 0, −1, 1, −4, −2, −10, 1, 2, 1, 1, 0,0, 3, 6, 0, 0, 0, −1, 0, 1, −2, −4, 1, 1 }, { 3, 7, 18, −6, −12, 3, 1,1, 2, 6, 19, −10, −15, 2, 0, 1, 3, 11, 34, −13, −32, 3, 0, 1, 15, 31,84, −21, −45, −6, −4, −1, 53, 95, 189, −17, −54, −5, −2, 0, −3, −3, −4,8, 6, 1, 1, 1, 3, 5, 8, −2, 1, 2, 1, 1, 0, −4, −1, 4, − 1, 1, 1, 0 }, {4, 0, −1, 0, 0, 0, −1, 0, 5, −1, −5, 1, 3, −1, 0, 0, 5, 0, −1, 0, 2, 0,−1, 0, −14, 7, 11, −8, 6, −7, −2, 0, 45, −5, −18, 8, 0, −1, −2, 1, 166,−71, −152, 54, 24, −5, −2, 1, −39, 28, 21, −16, −3, 1, 1, −1, −5, 3, 9,−4, −3, 1, 0, 0 }, { 0, 0, 0, 2, −2, 13, −2, −1, 1, 1, 1, 2, 22, −9, 2,0, 2, 0, 2, 1, 13, 37, −21, −3, 5, 0, 9, 12, 102, −133, −40, 5, −8, 16,−19, 87, −140, −44, 25, −2, 5, −9, 15, −31, 11, 25, −5, −2, −1, 5, −4,4, 1, −5, 2, 4, −1, 0, 1, 0, −5, 3, 4, −1 }, { −2, −2, 2, −9, 14, −6,−151, 109, 0, 0, 0, 2, −3, −7, 41, −23, 2, −1, 1, 3, −8, −3, 136, −85,−1, 1, −1, 2, 5, 3, −37, 23, −1, 1, −1, 2, −8, 6, −15, 13, 0, −1, 1, −3,6, −1, 4, −4, 0, −1, 0, 0, 0, −1, 1, 0, 0, 0, 0, 0, 0, 0, −1, 0 }, { −3,4, −7, 18, −24, 12, −6, 2, −4, 7, −12, 33, −47, 24, −8, 3, −8, 10, −24,55, −89, 38, −5, 1, −14, 23, −41, 108, −134, 27, 0, −6, −6, 24, −28, 99,−44, −22, 13, −3, 1, 4, −7, −2, 16, −8, 0, 0, 0, 2, −1, 0, 3, 0, 1, 0,−1, 1, −1, 1, −1, −1, 2, 0 }, { −1, −2, −2, 15, −17, −105, 54, 13, −1,−4, −2, 19, −21, −157, 65, 13, −1, −3, −1, 16, −15, −129, 10, 8, 0, 5,−1, 26, −8, −8, −38, −2, 0, 6, −2, 21, −14, 14, −12, −8, 1, 0, 0, −5, 5,10, 0, −2, 1, 0, 0, −1, 1, 3, 0, 0, 1, −1, 0, −1, 0, 2, 0, 0 }, { 2, 1,−1, −2, −1, 0, 0, −1, 3, 5, 0, −2, −2, 1, 0, 1, 2, 1, −2, −2, −2, 0, 1,0, −15, −15, −7, 2, −1, 3, 0, 1, 22, 24, −4, −16, −5, 0, 1, 0, 148, 180,60, −53, −29, −3, −1, −1, −25, −29, 5, 15, 4, 0, 0, 1, −14, −14, −4, 6,5, 0, 0, 0 }, { 1, 1, −1, 4, −7, 1, 72, −72, −1, −1, 0, −5, 9, −3, −121,118, 2, −1, 0, 2, −6, −10, 119, −95, 1, 0, 0, 1, 7, 7, −21, 4, −3, 3,−4, 4, −16, 3, −30, 33, 1, −3, 1, −4, 8, 2, 14, −15, 0, −1, 0, −2, 2,−3, −5, 5, 0, 0, −1, 2, −1, 1, 3, −3 }, { 1, 0, −1, −12, −12, 3, 2, 2,0, −1, −2, −18, −23, −1, 3, 3, −1, −3, −6, −32, −44, −11, −6, −3, −10,−23, −30, −88, −113, −62, −21, −5, −17, −39, −50, −132, −105, −24, −4,−4, −6, −11, −17, −28, −12, −3, 1, −1, −1, −1, 0, −6, −3, 2, 2, 0, 0, 3,1, 0, −3, 1, 0, −1 }, { −1, 2, −1, 0, −1, 1, 0, −1, −2, 3, −4, −1, 6,−3, 2, 1, −2, 2, 0, −1, −1, 2, −2, −1, 10, −14, 13, −8, 14, −13, −2, 0,−22, 34, −20, 3, −3, −2, 0, −1, −88, 138, −155, 37, 48, −19, 8, −5, 39,−55, 49, −9, −14, 7, −3, 1, 9, −10, 9, 0, −9, 3, 0, 0 }, { 1, 1, 1, 3,6, 12, 0, −1, 0, 0, −1, 2, −12, −12, −4, 4, −1, 2, −1, 5, −6, 19, 5, −4,−5, −9, −10, 1, −62, −180, −9, 5, 11, 6, 22, 23, 144, −5, −52, 3, −2, 2,−7, −14, −18, 45, 13, −11, 1, −1, 2, 11, −1, −14, 5, 4, 1, −1, 0, −5, 4,4, −1, 2 }, { −1, 0, 2, −1, 0, 1, 0, 0, −2, 0, 0, 0, 1, 0, 0, −1, 0, 0,1, −1, −1, 0, 0, −1, −6, 1, 2, 0, 0, −1, 0, −1, 11, −2, −9, 3, −1, 4, 2,0, −25, 3, 19, −3, 2, 5, 1, −1, −118, 30, 126, −28, −29, 0, 1, 1, 121,−34, −120, 32, 27, −4, −1, 0 }, { −3, −3, −2, −1, 0, −8, −73, −122, 0,0, 0, −1, −1, 2, 32, 24, 3, 4, 2, 1, 0, 15, 109, 162, 0, 0, 0, 2, 4, 3,−36, −32, 0, −1, −2, −2, −6, −4, −24, −47, 0, −1, −1, −3, 6, 8, 7, 6, 0,1, 0, −1, 3, 3, 2, 4, 1, −1, 1, 0, −1, 0, −1, 0 }, { 3, 4, 2, 1, 3, 46,129, 82, 3, 4, 2, −1, 5, 59, 148, 79, 1, 2, 1, 0, 8, 48, 58, 19, −1, −1,−1, −3, 5, 10, −30, −17, −1, −1, −1, −3, 3, −4, −22, −15, 0, −1, 1, 0,3, −1, −2, −2, 0, 0, 0, 0, 1, 0, −1, 0, 0, 0, 0, 1, 0, 0, −1, 0 }, { −1,−1, −1, −2, −1, 0, 0, −1, 1, 0, 3, 5, 1, −3, 0, 1, −1, −2, −1, −5, −3,−1, 0, −1, −2, −7, −8, −20, −14, −35, −11, −2, −4, −11, −7, −1, −10,−13, −4, 0, 16, 33, 92, 213, 73, −16, −6, 2, −4, −4, −19, −18, 8, 14, 3,−1, 1, −7, −15, −26, −11, 4, 2, 0 }, { −1, −2, −1, 0, 0, 0, 0, 0, 0, 0,0, 0, 0, 1, 1, 0, 1, 0, −1, 0, 0, −1, 0, 0, −4, −4, −1, 2, 2, 1, 1, 1,10, 11, 4, 0, −2, −3, −3, −1, −8, −11, −7, −1, 0, −2, 0, −1, −104, −132,−60, 29, 31, 6, 0, 0, 105, 128, 47, −34, −26, −3, 0, −1 }, { 0, 0, 0,−1, −2, 10, −9, 0, 0, 0, −1, −1, 2, −12, 14, 3, 0, −1, 0, −2, −4, 16,−10, 6, 0, −1, −1, −3, 3, −34, 93, −21, −6, 11, −12, 17, −43, 156, −113,18, 8, −10, 17, −33, 70, −80, 26, 2, −1, 1, −4, 2, −12, 17, 7, −8, −2,1, 1, −2, 2, 5, −14, 4 }, { −3, −4, −2, −2, −1, −16, −90, −100, 3, 4, 3,2, 1, 24, 116, 127, −2, −2, 0, 1, 0, −10, −83, −83, 0, −1, −1, −2, −4,0, 4, −9, 2, 0, 4, 1, 11, 1, 27, 43, −1, −1, 0, 1, −5, −3, −14, −15, 0,0, 1, 1, −1, 0, 5, 3, 0, 1, 0, 0, 1, 0, −2, −2 }, { 0, 0, −1, 2, −2, 5,−1, −2, −1, 0, −1, 2, −7, 2, 3, 1, 0, 1, 0, 1, 1, 8, −1, 3, 0, −1, −1,−5, −1, 7, 1, 11, −4, 6, −9, 22, −48, 86, −20, −4, −12, 12, −25, 87,−173, 47, 45, −22, −3, 3, −10, 65, −43, −58, 24, 5, 2, −1, −5, −7, 26,−20, −1, 9 }, { 0, 1, −1, 0, 1, −1, 0, 0, −1, 0, 0, 0, −1, 0, 0, 1, 0,0, −1, 0, 1, 0, 0, −1, −3, 3, −2, 0, −1, 1, 0, 0, 5, −6, 3, −2, 1, −2,−1, 1, −20, 27, −15, −4, 6, −7, 2, 0, −58, 83, −84, 19, 14, −7, 6, −4,101, −143, 116, −7, −38, 17, −6, 6 }, { 0, 1, 0, 0, 1, 7, −5, −9, 0, 1,0, 0, 2, 12, −16, −16, −1, 2, −1, −1, 3, 27, −73, 6, −4, 4, −4, −1, 0,44, −195, 71, −2, 3, 0, −5, 8, 43, −89, 27, 3, −2, 5, −14, 37, 5, 24,−13, 1, −1, 2, −3, 8, −10, 9, 2, −1, 0, −1, −1, −3, 2, 1, 1 }, { −3, −1,3, 0, −1, 0, 1, −1, −4, −1, 4, 0, −1, 0, 0, 0, −2, −1, 2, 1, 0, 0, 1, 0,0, −2, −1, 1, −2, 0, −1, 1, −1, −5, −1, 3, 0, 2, 2, 0, −27, −6, 34, 8,−8, 1, 6, −2, −99, 4, 137, −6, −36, 2, 4, 2, −101, 15, 144, −12, −45, 7,4, 1 }, { 1, 0, −1, 4, −4, 15, 60, −126, 1, −1, 0, 4, −4, 15, 85, −160,1, 0, 0, 2, −2, 9, 57, −84, 0, 1, −1, 2, −3, −2, 4, 32, −1, 0, −1, 1,−4, −4, −3, 30, 0, −1, 0, 0, 0, −7, −2, 2, 2, 0, −1, 1, −2, −1, −1, 2,0, −1, 0, 0, 0, 0, −2, 1 }, { 0, 0, 0, −1, 4, 2, −3, 0, 0, 0, 0, 1, −5,−1, 5, −1, 0, 0, −1, −1, 5, 3, −5, 0, 1, 0, 1, −2, 0, −4, 15, 5, −2, 3,−6, 14, −10, 21, −26, 0, −4, 4, −2, 17, −72, 5, 18, −14, 7, −8, 12,−132, 156, −6, −30, 13, 0, −1, 0, 83, −86, 15, 10, −9 }, { 0, 0, 0, 1,1, −3, −5, 1, 0, 0, 1, 2, 0, −6, −2, 2, 0, 0, 0, 2, 6, −5, −12, −5, 1,1, 3, 6, 14, −6, −18, −21, 0, −1, 5, 16, 6, −103, −79, 14, −5, −4, −4,13, −98, −148, 36, 52, −2, 1, −8, −23, −32, 83, 29, −14, 2, 1, 3, −1,33, 21, −7, −3 }, { 0, 1, 1, 0, 2, 10, 12, 1, −1, −1, −1, −1, −4, −13,−14, −5, 1, 2, 2, 1, 4, 21, 23, 12, −3, −4, −4, −6, −15, −41, −85, −49,5, 6, 4, 6, 26, 109, 150, 43, −3, −4, −1, −1, −32, −104, −36, 23, −1, 0,−2, −7, 7, 24, −19, −19, 1, 3, 1, 3, 1, 4, 9, 1 }, { −3, 4, −1, −1, 1,−2, 1, −1, −4, 4, −1, −2, 2, 0, 1, 0, −2, 2, 1, −2, 2, −2, 1, −1, 1, −1,3, −2, 3, −2, 1, 0, −5, 6, 0, −3, 2, −3, −2, 1, −46, 60, −34, −4, 13,−6, 0, 0, −108, 140, −80, −14, 37, −13, 6, −6, −78, 96, −46, −15, 31,−12, 4, −4 }, { 0, 1, 1, 2, 3, 0, −1, 0, 0, −1, −1, −3, −3, −2, 0, 1, 1,1, 1, 3, 3, 0, −1, −1, 0, 0, 1, 1, 1, 3, 4, 1, 0, −1, −1, −2, −3, −11,−12, −1, −3, −6, −8, −7, −18, −27, −8, −2, 11, 24, 61, 123, 116, 32, 2,0, −9, −22, −59, −134, −89, 0, 7, 4 }, { 0, 0, 0, 0, 0, −5, 0, 19, 0, 0,0, 0, 1, 6, −2, −28, 0, 0, 0, 0, 0, −9, 2, 39, 0, −1, 1, 0, 5, 14, −5,−135, −2, 1, −2, −1, −6, −37, −28, 191, 0, 1, 0, 1, 6, 38, 11, −51, 1,−2, 2, 3, 1, −34, 0, −6, 0, 0, −2, −4, −5, 7, −2, 12 }, { 3, 3, 1, −1,−1, −1, 0, 0, 3, 3, 1, −1, −1, 0, −1, 1, 1, 1, 0, 0, −1, 0, 1, 0, −1,−2, −1, −1, 0, 0, 0, 0, −3, −4, 0, 0, 0, 1, 0, 0, 23, 29, 20, 2, −4, −1,2, 0, 106, 121, 61, −8, −17, −3, 2, 2, 113, 127, 65, −11, −20, −2, 1, 0}, { 0, 0, 0, 0, 0, 1, 1, −1, 0, 0, 0, 0, 0, 1, 3, −3, 0, 0, 0, 0, 0, 1,0, 4, 0, 0, 0, 0, 0, −1, −1, 17, 1, −1, 2, −1, 3, −7, 43, −22, 4, −4, 4,−2, 4, −4, 105, −136, −2, 6, −6, 3, −9, 66, −57, −42, −5, 8, −8, −1, 0,57, −126, 66 }, { 0, −1, 0, 2, 1, −1, −1, 0, −1, −1, 0, 2, 1, −2, −1,−1, 0, −1, 0, 0, 0, −2, −1, −1, −1, −2, −1, −3, −1, −4, −1, −3, −2, −4,−4, −4, −3, −12, −10, 3, −3, −6, 0, 25, 23, −14, −14, 5, 0, 6, 36, 132,104, 13, −16, −5, 1, 9, 43, 139, 110, 5, −12, −4 }, { 0, 0, 0, 0, −1,−1, 4, 9, 0, 0, −1, 0, −2, −4, 10, 30, 0, 0, −1, 0, −3, −7, 24, 80, 0,1, −1, 0, −5, −13, 51, 187, −1, 0, −1, 0, −5, −15, 27, 130, −1, −1, 0,3, −8, −27, −14, 5, 0, −1, 1, 0, −3, −7, −8, −17, 1, −1, 0, −1, 4, −2,−1, −4 }, { 0, −1, 1, −1, 1, −4, 2, 0, 1, −1, 1, −1, 2, −5, 6, 1, 1, −1,1, −1, 1, −4, 1, 0, 1, −1, 1, −1, 2, −6, 4, 2, 2, −4, 4, −4, 11, −31,42, −20, 7, −10, 11, −13, 34, −87, 119, −35, 8, −14, 17, −16, 44, −140,78, 17, 4, −10, 10, −20, 32, −78, 21, 27 }, { 0, 0, −1, 2, 0, −2, 0, 0,0, 0, −1, 2, −1, −2, 2, 1, 0, 0, −1, 1, 1, −3, 0, 2, 0, 0, 0, 0, 1, −3,−3, 1, 1, −1, 2, −5, 12, −19, 10, −3, 3, −2, −1, 4, 18, −37, 37, −3, −7,13, −28, 83, −74, −22, 21, 9, −14, 23, −48, 128, −163, 14, 34, −5 }, {0, 0, 1, 1, 0, −3, −4, −3, 0, 0, 1, 1, 0, −3, −6, −3, 1, 1, 1, 2, 2, −3,−4, −2, 1, 1, 2, 2, 3, −5, −8, −6, 0, 0, 3, 4, 0, −28, −50, −29, −2, −1,1, 5, −19, −74, −120, −67, −2, −2, 2, 4, −23, −101, −110, −55, 0, −2, 0,1, −21, −73, −65, −22 }, { 0, 0, 0, 0, 0, 2, 5, 6, 0, 0, −1, 0, −1, −3,−8, −7, 0, 0, 0, 0, 1, 1, 6, 10, 0, 0, 0, 0, 0, −1, 0, 7, 0, 1, 1, 1, 2,5, 4, −1, −2, −1, −2, −1, −8, −24, −82, −112, 0, 1, −4, −6, 1, 40, 126,138, −1, 0, 2, 11, −1, −12, −65, −65 }, { 0, 0, 0, 0, 0, −3, −1, 1, 0,0, 0, −1, 1, 3, 2, 0, 0, 0, 0, 0, 0, −4, −2, 1, 0, 0, 1, 0, 0, 2, 2, 1,0, −1, 1, −1, 0, −4, 4, −9, 1, −3, 2, 1, 2, −33, 16, 17, −1, −2, 5, 9,−3, −103, −54, 73, 0, 7, −8, −30, 24, 191, −10, −82 }, { 0, 0, 0, 0, 0,1, −4, 3, 0, 0, 0, 0, 0, −2, 7, −5, 0, 0, 0, 0, 0, 1, −6, 7, 0, 0, 0, 0,0, −2, 3, 1, 1, −1, 1, −1, 2, −3, 1, 5, 2, −2, 2, 0, 0, −13, 63, −61,−6, 8, −9, 11, −20, 61, −136, 109, 4, −9, 9, −12, 21, −75, 110, −64 }, {0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0, 0, 0, 0, 0, −2,−3, 0, −1, 0, 0, −1, −2, −6, −12, 0, 0, 0, −1, 1, 4, 18, 24, 1, 2, 1, 1,10, 29, 74, 106, 0, 2, 1, 6, 4, 4, −21, 16, −3, −3, −1, 0, −24, −80,−150, −128 }, { 0, 0, 0, 0, 0, 1, 1, −2, 0, 0, 0, 0, 0, 1, 2, −5, 0, 0,0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 4, −1, 1, 0, 0, 0, 1, 3, 7, −14, 2,−2, 1, −2, 1, 9, 21, −78, 4, −3, 3, −4, 5, 21, 53, −150, 3, −4, 4, −4,10, 19, 64, −168 },

In the above example, each transform coefficient is represented by 9bits (i.e., 1 bit: sign, 8 bits: absolute values of 0 to 255). In anembodiment of the present disclosure, various precisions may be used torepresent the transform coefficients. For example, instead of 9 bits, 8bits may be used to represent each coefficient. In this case, a sign bitis not changed, but ranges of the absolute values may be changed.

The embodiments of the present disclosure descried above have beendescribed separately for the convenience of description, but the presentdisclosure is not limited thereto. That is, the embodiments described inEmbodiments 1 to 14 described above may be performed independently andone or more various embodiments may be combined and performed.

FIG. 30 is a diagram illustrating an inverse transform method accordingto an embodiment to which the present disclosure is applied.

Referring to FIG. 30, the decoder is mainly described for convenience ofdescription, but the method for performing the transform/inversetransform based on the secondary transform set according to anembodiment of the present disclosure may be substantially equallyperformed even in the encoder.

The decoder generates a dequantized transform block by performingdequantization for a current block (S3001).

The decoder acquires an intra prediction mode of the current block(S3002).

The decoder determines a secondary transform set applied to the currentblock among a plurality of secondary transform sets based on the intraprediction mode (S3003).

As described above, the plurality of secondary transform sets mayinclude at least one mixed secondary transform set.

Furthermore, as described above, the mixed secondary transform set mayinclude at least one 8×8 transform kernel applied to an 8×8 size areaand at least one 4×4 transform kernel applied to a 4×4 size area.

In addition, as described above, when the plurality of secondarytransform sets include a plurality of mixed secondary transform sets,the plurality of mixed secondary transform sets may include differentnumbers of transform kernels, respectively.

Furthermore, as described above, step S3003 may include determiningwhether to use the mixed secondary transform set based on the intraprediction mode, and determining, based on a size of the current block,the secondary transform set applied to the current block among theplurality of secondary transform sets including the at least one mixedsecondary transform set when it is determined that the mixed secondarytransform set is used and determining, based on the size of the currentblock, the secondary transform set applied to the current block amongresidual secondary transform sets other than the at least one mixedsecondary transform set when it is determined that the mixed secondarytransform set is not used.

The decoder derives a transform kernel applied to the current block inthe determined secondary transform set (S3004).

As described above, step S3004 may further include acquiring a secondarytransform index indicating a transform kernel applied to the currentblock in the determined secondary transform set. As an embodiment, thesecondary transform index may be binarized by a truncated unary schemebased on a maximum number of available transform kernels in thedetermined secondary transform set.

The decoder performs secondary transform for a top left specific area ofthe dequantized transform block by using the derived transform kernel(S3005).

FIG. 31 is a diagram illustrating a dequantization unit and an inversetransform unit according to an embodiment to which the presentdisclosure is applied.

In FIG. 31, the inverse transform unit 3100 is illustrated as one blockfor convenience of description, but the inter prediction unit may beimplemented as a component included in the encoder and/or the decoder.

Referring to FIG. 31, a dequantization unit 3101 and an inversetransform unit 3100 implement the functions, procedures, and/or methodsproposed in FIGS. 4 to 30 above. Specifically, the inverse transformunit 3100 may be configured to include an intra prediction modeacquiring unit 3102, a secondary transform set determining unit 3103, atransform kernel deriving unit 3104, and a secondary inverse transformunit 3105.

The dequantization unit 3101 generates a dequantized transform block byperforming dequantization for a current block.

The intra prediction mode acquiring unit 3102 acquires an intraprediction mode of the current block.

The secondary transform set determining unit determines a secondarytransform set applied to the current block among a plurality ofsecondary transform sets based on the intra prediction mode.

As described above, the plurality of secondary transform sets mayinclude at least one mixed secondary transform set.

Furthermore, as described above, the mixed secondary transform set mayinclude at least one 8×8 transform kernel applied to an 8×8 size areaand at least one 4×4 transform kernel applied to a 4×4 size area.

In addition, as described above, when the plurality of secondarytransform sets include a plurality of mixed secondary transform sets,the plurality of mixed secondary transform sets may include differentnumbers of transform kernels, respectively.

Furthermore, as described above, the secondary transform set determiningunit 3103 may determine whether to use the mixed secondary transform setbased on the intra prediction mode, and determine, based on a size ofthe current block, the secondary transform set applied to the currentblock among the plurality of secondary transform sets including the atleast one mixed secondary transform set when it is determined that themixed secondary transform set is used and determining, based on the sizeof the current block, the secondary transform set applied to the currentblock among residual secondary transform sets other than the at leastone mixed secondary transform set when it is determined that the mixedsecondary transform set is not used.

The transform kernel deriving unit 3104 derives a transform kernelapplied to the current block in the determined secondary transform set.

As described above, the transform kernel deriving unit 3104 may acquirea secondary transform index indicating a transform kernel applied to thecurrent block in the determined secondary transform set. As anembodiment, the secondary transform index may be binarized by atruncated unary scheme based on a maximum number of available transformkernels in the determined secondary transform set.

The secondary inverse transform unit 3105 performs secondary inversetransform for a top left specific area of the dequantized transformblock by using the derived transform kernel.

The plurality of secondary transform sets include at least one mixedsecondary transform set.

FIG. 32 illustrates a video coding system to which the presentdisclosure is applied.

The video coding system may include a source device and a receivingdevice. The source device may transfer encoded video/image informationor data to the receiving device through a digital storage medium ornetwork in a file or streaming form.

The source device may include a video source, an encoding apparatus, anda transmitter. The receiving device may include a receiver, a decodingapparatus, and a renderer. The encoding apparatus may be called avideo/image encoding apparatus and the decoding apparatus may be calleda video/image decoding apparatus. The transmitter may be included in theencoding apparatus. The receiver may be included in the decodingapparatus. The renderer may include a display and the display may beconfigured as a separate device or an external component.

A video source may acquire a video/image through a capturing,synthesizing, or generating process of the video/image. The video sourcemay include a video/image capture device and/or a video/image generationdevice. The video/image capture device may include, for example, one ormore cameras, video/image archives including previously capturedvideo/images, and the like. The video/image generation device mayinclude, for example, a computer, a tablet, and a smart phone and may(electronically) generate the video/image. For example, a virtualvideo/image may be generated through the computer, etc., and in thiscase, the video/image capturing process may be replaced by a process ofgenerating related data.

The encoding apparatus may encode an input video/image. The encodingapparatus may perform a series of procedures including prediction,transform, quantization, and the like for compression and codingefficiency. The encoded data (encoded video/image information) may beoutput in the bitstream form.

The transmitter may transfer the encoded video/image information or dataoutput in the bitstream from to the receiver of the receiving devicethrough the digital storage medium or network in the file or streamingform. The digital storage medium may include various storage media suchas USB, SD, CD, DVD, Blu-ray, HDD, SSD, and the like. The transmittermay include an element for generating a media file through apredetermined file format and may include an element for transmissionthrough a broadcast/communication network. The receiver may extract thebitstream and transfer the extracted bitstream to the decodingapparatus.

The decoding apparatus performs a series of procedures includingdequantization, inverse transform, prediction, etc., corresponding to anoperation of the encoding apparatus to decode the video/image.

The renderer may render a decoded video/image. The rendered video/imagemay be displayed through the display.

FIG. 33 is a structure diagram of a content streaming system as anembodiment to which the present disclosure is applied.

Referring to FIG. 33, the content streaming system to which the presentdisclosure is applied may largely include an encoding server, astreaming server, a web server, a media storage, a user device, and amultimedia input device.

The encoding server compresses contents input from multimedia inputdevices including a smartphone, a camera, a camcorder, etc., intodigital data to serve to generate the bitstream and transmit thebitstream to the streaming server. As another example, when themultimedia input devices including the smartphone, the camera, thecamcorder, etc., directly generate the bitstream, the encoding servermay be omitted.

The bitstream may be generated by the encoding method or the bitstreamgenerating method to which the present disclosure is applied and thestreaming server may temporarily store the bitstream in the process oftransmitting or receiving the bitstream.

The streaming server transmits multimedia data to the user device basedon a user request through a web server, and the web server serves as anintermediary for informing a user of what service there is. When theuser requests a desired service to the web server, the web servertransfers the requested service to the streaming server and thestreaming server transmits the multimedia data to the user. In thiscase, the content streaming system may include a separate control serverand in this case, the control server serves to control acommand/response between respective devices in the content streamingsystem.

The streaming server may receive contents from the media storage and/orthe encoding server. For example, when the streaming server receives thecontents from the encoding server, the streaming server may receive thecontents in real time. In this case, the streaming server may store thebitstream for a predetermined time in order to provide a smoothstreaming service.

Examples of the user device may include a cellular phone, a smart phone,a laptop computer, a digital broadcasting terminal, a personal digitalassistants (PDA), a portable multimedia player (PMP), a navigation, aslate PC, a tablet PC, an ultrabook, a wearable device such as asmartwatch, a smart glass, or a head mounted display (HMD), etc., andthe like.

Each server in the content streaming system may be operated as adistributed server and in this case, data received by each server may bedistributed and processed.

As described above, the embodiments described in the present disclosuremay be implemented and performed on a processor, a microprocessor, acontroller, or a chip. For example, functional units illustrated in eachdrawing may be implemented and performed on a computer, the processor,the microprocessor, the controller, or the chip.

In addition, the decoder and the encoder to which the present disclosuremay be included in a multimedia broadcasting transmitting and receivingdevice, a mobile communication terminal, a home cinema video device, adigital cinema video device, a surveillance camera, a video chat device,a real time communication device such as video communication, a mobilestreaming device, storage media, a camcorder, a video on demand (VoD)service providing device, an (Over the top) OTT video device, anInternet streaming service providing devices, a 3 dimensional (3D) videodevice, a video telephone video device, a transportation means terminal(e.g., a vehicle terminal, an airplane terminal, a ship terminal, etc.),and a medical video device, etc., and may be used to process a videosignal or a data signal. For example, the Over the top (OTT) videodevice may include a game console, a Blu-ray player, an Internet accessTV, a home theater system, a smartphone, a tablet PC, a digital videorecorder (DVR), and the like.

In addition, a processing method to which the present disclosure isapplied may be produced in the form of a program executed by thecomputer, and may be stored in a computer-readable recording medium.Multimedia data having a data structure according to the presentdisclosure may also be stored in the computer-readable recording medium.The computer-readable recording medium includes all types of storagedevices and distribution storage devices storing computer-readable data.The computer-readable recording medium may include, for example, aBlu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, anEPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, andan optical data storage device. Further, the computer-readable recordingmedium includes media implemented in the form of a carrier wave (e.g.,transmission over the Internet). Further, the bitstream generated by theencoding method may be stored in the computer-readable recording mediumor transmitted through a wired/wireless communication network.

In addition, the embodiment of the present disclosure may be implementedas a computer program product by a program code, which may be performedon the computer by the embodiment of the present disclosure. The programcode may be stored on a computer-readable carrier.

In the embodiments described above, the components and the features ofthe present disclosure are combined in a predetermined form. Eachcomponent or feature should be considered as an option unless otherwiseexpressly stated. Each component or feature may be implemented not to beassociated with other components or features. Further, the embodiment ofthe present disclosure may be configured by associating some componentsand/or features. The order of the operations described in theembodiments of the present disclosure may be changed. Some components orfeatures of any embodiment may be included in another embodiment orreplaced with the component and the feature corresponding to anotherembodiment. It is apparent that the claims that are not expressly citedin the claims are combined to form an embodiment or be included in a newclaim by an amendment after the application.

The embodiments of the present disclosure may be implemented byhardware, firmware, software, or combinations thereof. In the case ofimplementation by hardware, according to hardware implementation, theexemplary embodiment described herein may be implemented by using one ormore application specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), field programmable gate arrays(FPGAs), processors, controllers, micro-controllers, microprocessors,and the like.

In the case of implementation by firmware or software, the embodiment ofthe present disclosure may be implemented in the form of a module, aprocedure, a function, and the like to perform the functions oroperations described above. A software code may be stored in the memoryand executed by the processor. The memory may be positioned inside oroutside the processor and may transmit and receive data to/from theprocessor by already various means.

It is apparent to those skilled in the art that the present disclosuremay be embodied in other specific forms without departing from essentialcharacteristics of the present disclosure. Accordingly, theaforementioned detailed description should not be construed asrestrictive in all terms and should be exemplarily considered. The scopeof the present disclosure should be determined by rational construing ofthe appended claims and all modifications within an equivalent scope ofthe present disclosure are included in the scope of the presentdisclosure.

INDUSTRIAL APPLICABILITY

Hereinabove, the preferred embodiments of the present disclosure aredisclosed for an illustrative purpose and hereinafter, modifications,changes, substitutions, or additions of various other embodiments willbe made within the technical spirit and the technical scope of thepresent disclosure disclosed in the appended claims by those skilled inthe art.

The invention claimed is:
 1. A method for decoding a video signal, themethod comprising: generating, by a processor, a dequantized transformblock for a current block; acquiring, by the processor, an intraprediction mode of the current block; determining, by the processor, asecondary transform set applied to the current block among a pluralityof secondary transform sets based on the intra prediction mode;deriving, by the processor, a transform kernel applied to the currentblock in the determined secondary transform set; and performing, by theprocessor, secondary transform for some coefficients of the dequantizedtransform block based on the derived transform kernel, wherein thederiving, by the processor, the transform kernel further includes:acquiring a secondary transform index including information about thetransform kernel applied to the current block in the determinedsecondary transform set, and wherein the secondary transform index isbinarized in a truncated unary scheme based on a number of availabletransform kernels in the determined secondary transform set.
 2. Themethod of claim 1, wherein the plurality of secondary transform setsinclude at least one mixed secondary transform set.
 3. The method ofclaim 2, wherein the mixed secondary transform set includes at least one8×8 transform kernel applied to an 8×8 size area and at least one 4×4transform kernel applied to a 4×4 size area.
 4. The method of claim 3,wherein when the plurality of secondary transform sets include aplurality of mixed secondary transform sets, the plurality of mixedsecondary transform sets include different numbers of transform kernels,respectively.
 5. The method of claim 4, wherein the secondary transformindex is binarized in the truncated unary scheme based on a maximumnumber of available transform kernels in the determined secondarytransform set.
 6. The method of claim 2, wherein the determining of thesecondary transform set includes: determining whether to use the mixedsecondary transform set based on the intra prediction mode, anddetermining, based on a size of the current block, the secondarytransform set applied to the current block among the plurality ofsecondary transform sets including the at least one mixed secondarytransform set when it is determined that the mixed secondary transformset is used and determining, based on the size of the current block, thesecondary transform set applied to the current block among residualsecondary transform sets other than the at least one mixed secondarytransform set when it is determined that the mixed secondary transformset is not used.
 7. The method of claim 1, wherein each secondarytransform set of the plurality of secondary transform sets includes 2transform kernels.
 8. A method for encoding a video signal, comprising:performing, by a processor, intra prediction for a current block toobtain a residual block for the current block; performing, by theprocessor, primary transform for the residual block to obtain aprimary-transformed block; determining, by the processor, a secondarytransform set applied to the primary-transformed block among a pluralityof secondary transform sets based on an intra prediction mode applied tothe current block; deriving, by the processor, a transform kernelapplied to the current block in the determined secondary transform set;performing, by the processor, secondary transform for some coefficientsof the primary-transformed block based on the derived transform kernel;and encoding, by the processor, a secondary transform index includinginformation about the transform kernel, wherein the secondary transformindex is binarized in a truncated unary scheme based on a number ofavailable transform kernels in the determined secondary transform set.9. The method of claim 8, wherein the plurality of secondary transformsets include at least one mixed secondary transform set.
 10. The methodof claim 9, wherein the mixed secondary transform set includes at leastone 8×8 transform kernel applied to an 8×8 size area and at least one4×4 transform kernel applied to a 4×4 size area.
 11. The method of claim8, wherein each secondary transform set of the plurality of secondarytransform sets includes 2 transform kernels.
 12. A non-transitorydecoder-readable storage medium for storing a bitstream generated by theencoding method according to claim 8.